2025-04-02 02:52:17,032 [ 185854 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-04-02 02:52:17,032 [ 185854 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:97, check_args_and_update_paths) 2025-04-02 02:52:17,032 [ 185854 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:108, check_args_and_update_paths) 2025-04-02 02:52:17,032 [ 185854 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:110, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_xo8ksw --privileged --dns-search='.' --memory=30709018624 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=8b2301119731 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_access_for_functions/test.py::test_access_rights_for_function test_access_for_functions/test.py::test_ignore_obsolete_grant_on_database test_aggregation_memory_efficient/test.py::test_remote test_allowed_client_hosts/test.py::test_allowed_host test_allowed_client_hosts/test.py::test_denied_host test_alter_codec/test.py::test_alter_codec_index test_alter_codec/test.py::test_alter_codec_pk test_alter_on_mixed_type_cluster/test.py::test_alter_on_cluter_non_replicated test_alter_on_mixed_type_cluster/test.py::test_alter_replicated_on_cluster test_alternative_keeper_config/test.py::test_create_insert test_async_connect_to_multiple_ips/test.py::test test_async_metrics_in_cgroup/test.py::test_normalized_user_cpu test_async_metrics_in_cgroup/test.py::test_system_wide_metrics test_async_metrics_in_cgroup/test.py::test_user_cpu_accounting 'test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Atomic]' 'test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Ordinary]' test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart test_authentication/test.py::test_authentication_fail test_authentication/test.py::test_authentication_pass test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication test_backup_restore/test.py::test_attach_partition test_backup_restore/test.py::test_replace_partition test_backup_restore/test.py::test_restore test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True]' test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes test_backup_restore_on_cluster/test.py::test_empty_replicated_table test_backup_restore_on_cluster/test.py::test_file_deduplication test_backup_restore_on_cluster/test.py::test_get_error_from_other_host test_backup_restore_on_cluster/test.py::test_keeper_value_max_size test_backup_restore_on_cluster/test.py::test_mutation test_backup_restore_on_cluster/test.py::test_projection test_backup_restore_on_cluster/test.py::test_replicated_database test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 'test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[database]' 'test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[uuid]' test_backup_restore_on_cluster/test.py::test_replicated_table test_backup_restore_on_cluster/test.py::test_replicated_table_after_alters test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath test_backup_restore_on_cluster/test.py::test_required_privileges test_backup_restore_on_cluster/test.py::test_shutdown_waits_for_backup test_backup_restore_on_cluster/test.py::test_system_functions test_backup_restore_on_cluster/test.py::test_system_users test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty test_backup_restore_on_cluster/test.py::test_tables_dependency test_backward_compatibility/test_convert_ordinary.py::test_convert_ordinary_to_atomic test_backward_compatibility/test_cte_distributed.py::test_cte_distributed test_backward_compatibility/test_functions.py::test_aggregate_states test_backward_compatibility/test_functions.py::test_string_functions test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column test_backward_compatibility/test_select_aggregate_alias_column.py::test_select_aggregate_alias_column test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts test_broken_part_during_merge/test.py::test_merge_and_part_corruption test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile test_build_sets_from_multiple_threads/test.py::test_set test_cancel_freeze/test.py::test_cancel_backup test_cgroup_limit/test.py::test_cgroup_cpu_limit test_check_table/test.py::test_check_all_tables 'test_check_table/test.py::test_check_normal_table_corruption[]' 'test_check_table/test.py::test_check_replicated_table_corruption[-_0-.bin]' 'test_check_table/test.py::test_check_replicated_table_simple[-_0]' test_cleanup_after_start/test.py::test_old_dirs_cleanup test_cluster_all_replicas/test.py::test_cluster 'test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[one_shard_three_nodes]' 'test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[two_shards_three_nodes]' test_cluster_all_replicas/test.py::test_global_in 'test_cluster_all_replicas/test.py::test_skip_unavailable_replica[one_shard_three_nodes]' 'test_cluster_all_replicas/test.py::test_skip_unavailable_replica[two_shards_three_nodes]' test_cluster_discovery/test_password.py::test_connect_with_password test_compatibility_merge_tree_settings/test.py::test_check_projections_compatibility test_compatibility_merge_tree_settings/test.py::test_config_overrides_compatibility test_compression_nested_columns/test.py::test_nested_compression_codec test_concurrent_queries_for_all_users_restriction/test.py::test_exception_message test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_1 test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_50 test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_limit_reached test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_default test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_soft_limit_defined_50 test_config_substitutions/test.py::test_allow_databases test_config_substitutions/test.py::test_config test_config_substitutions/test.py::test_config_from_env_overrides test_config_substitutions/test.py::test_config_merge_from_env_overrides test_config_substitutions/test.py::test_config_multiple_zk_substitutions test_config_substitutions/test.py::test_include_config test_custom_settings/test.py::test_custom_settings -vvv -ss" altinityinfra/integration-tests-runner:2165613c5fcd '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] test_cluster_all_replicas/test.py::test_cluster Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] test_config_substitutions/test.py::test_allow_databases test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default test_check_table/test.py::test_check_all_tables test_async_metrics_in_cgroup/test.py::test_normalized_user_cpu test_alter_codec/test.py::test_alter_codec_index test_backup_restore/test.py::test_attach_partition test_allowed_client_hosts/test.py::test_allowed_host test_access_for_functions/test.py::test_access_rights_for_function Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Stdout:1 Command:[docker network prune --force] No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Stdout:1 Command:[docker network prune --force] No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stdout:net.ipv4.ip_local_port_range = 55000 65535 Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_backup_restore/test.py Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Cluster start called. is_up=False Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/test.py Stdout:net.ipv4.ip_local_port_range = 55000 65535 Cluster start called. is_up=False Running tests in /ClickHouse/tests/integration/test_access_for_functions/test.py Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Cluster start called. is_up=False Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Running tests in /ClickHouse/tests/integration/test_allowed_client_hosts/test.py Cluster start called. is_up=False Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stdout:net.ipv4.ip_local_port_range = 55000 65535 Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Running tests in /ClickHouse/tests/integration/test_cluster_all_replicas/test.py Stdout:net.ipv4.ip_local_port_range = 55000 65535 Cluster start called. is_up=False Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_check_table/test.py Running tests in /ClickHouse/tests/integration/test_config_substitutions/test.py Cluster start called. is_up=False Cluster start called. is_up=False Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_alter_codec/test.py Cluster start called. is_up=False Stdout:net.ipv4.ip_local_port_range = 55000 65535 Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Running tests in /ClickHouse/tests/integration/test_async_metrics_in_cgroup/test.py Cluster start called. is_up=False Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_backup_restore_on_cluster/test.py Cluster start called. is_up=False Docker networks for project roottestconcurrentthreadssoftlimit-gw0 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestallowedclienthosts-gw8 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestaccessforfunctions-gw7 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestclusterallreplicas-gw1 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestbackuprestore-gw6 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestconfigsubstitutions-gw4 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestchecktable-gw3 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestasyncmetricsincgroup-gw5 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestaltercodec-gw9 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestbackuprestoreoncluster-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestallowedclienthosts-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestconcurrentthreadssoftlimit-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestclusterallreplicas-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestbackuprestore-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestconfigsubstitutions-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestasyncmetricsincgroup-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestchecktable-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestaltercodec-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestbackuprestoreoncluster-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestaccessforfunctions-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestallowedclienthosts-gw8 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottestbackuprestore-gw6 are DRIVER VOLUME NAME Docker volumes for project roottestclusterallreplicas-gw1 are DRIVER VOLUME NAME Docker volumes for project roottestconcurrentthreadssoftlimit-gw0 are DRIVER VOLUME NAME Cleanup called Cleanup called Cleanup called Docker volumes for project roottestchecktable-gw3 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottestasyncmetricsincgroup-gw5 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottestconfigsubstitutions-gw4 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottestaltercodec-gw9 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottestaccessforfunctions-gw7 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottestbackuprestoreoncluster-gw2 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestconfigsubstitutions-gw4 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestbackuprestore-gw6 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestaltercodec-gw9 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestchecktable-gw3 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestallowedclienthosts-gw8 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestasyncmetricsincgroup-gw5 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestconcurrentthreadssoftlimit-gw0 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestbackuprestoreoncluster-gw2 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestaccessforfunctions-gw7 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestclusterallreplicas-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestconfigsubstitutions-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestconcurrentthreadssoftlimit-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestallowedclienthosts-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestbackuprestore-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestaltercodec-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestchecktable-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestasyncmetricsincgroup-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestbackuprestoreoncluster-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestaccessforfunctions-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestclusterallreplicas-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestallowedclienthosts-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestallowedclienthosts-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestchecktable-gw3 are DRIVER VOLUME NAME Docker volumes for project roottestconcurrentthreadssoftlimit-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestchecktable-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestconcurrentthreadssoftlimit-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestaccessforfunctions-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestaccessforfunctions-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestasyncmetricsincgroup-gw5 are DRIVER VOLUME NAME Docker volumes for project roottestaltercodec-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestasyncmetricsincgroup-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestaltercodec-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestconfigsubstitutions-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestconfigsubstitutions-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestclusterallreplicas-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestclusterallreplicas-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestbackuprestore-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackuprestore-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestbackuprestoreoncluster-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackuprestoreoncluster-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestconcurrentthreadssoftlimit-gw0 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestbackuprestore-gw6 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestaccessforfunctions-gw7 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestasyncmetricsincgroup-gw5 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestclusterallreplicas-gw1 Trying to prune unused networks... Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestchecktable-gw3 Trying to prune unused networks... No running containers for project: roottestconfigsubstitutions-gw4 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestaltercodec-gw9 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestbackuprestoreoncluster-gw2 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Unstopped containers: {} No running containers for project: roottestallowedclienthosts-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Stderr:Error response from daemon: a prune operation is already running Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Exitcode:1 Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Trying to prune unused images... Command:[docker image prune -f] Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Setup directory for instance: instance Create directory for configuration generated in this helper Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Create directory for common tests configuration Stdout:1 Volumes pruned: 1 Setup directory for instance: node Copy common configuration from helpers Generate and write macros file Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Copy custom test config files [] to /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/configs/config.d Setup database dir /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/database Generate and write macros file Setup logs dir /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/logs Copy custom test config files [] to /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/database Setup logs dir /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/.env Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Stdout:Total reclaimed space: 0B Images pruned Generate and write macros file Trying to prune unused volumes... Command:[docker volume ls | wc -l] Copy custom test config files ['/ClickHouse/tests/integration/test_cluster_all_replicas/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/configs/config.d Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Stdout:1 Setup database dir /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/database Volumes pruned: 1 Setup logs dir /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/logs Setup directory for instance: node1 Create directory for configuration generated in this helper Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Create directory for common tests configuration Setup directory for instance: node2 Create directory for configuration generated in this helper Copy common configuration from helpers Create directory for common tests configuration Copy common configuration from helpers Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Stdout:1 Generate and write macros file Generate and write macros file Volumes pruned: 1 Setup directory for instance: node1 Copy custom test config files [] to /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/configs/config.d Generate and write macros file Stdout:1 Setup database dir /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/database Setup database dir /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/database Volumes pruned: 1 Setup logs dir /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/logs Copy custom test config files ['/ClickHouse/tests/integration/test_cluster_all_replicas/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/configs/config.d Setup directory for instance: node1 Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup logs dir /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/logs Setup database dir /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/database Setup directory for instance: node2 Create directory for configuration generated in this helper Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup logs dir /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/logs Create directory for common tests configuration Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/.env Create directory for configuration generated in this helper Create directory for configuration generated in this helper Copy common configuration from helpers Create directory for common tests configuration Create directory for common tests configuration Create directory for configuration generated in this helper Copy common configuration from helpers Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Copy common configuration from helpers Create directory for common tests configuration No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Copy common configuration from helpers Generate and write macros file Generate and write macros file Generate and write macros file No config file found Copy custom test config files ['/ClickHouse/tests/integration/test_alter_codec/configs/logs_config.xml'] to /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/configs/config.d Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_concurrent_threads_soft_limit/configs/config_default.xml'] to /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/database Copy custom test config files [] to /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/configs/config.d Setup logs dir /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/logs Setup database dir /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/database Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup logs dir /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/logs Setup database dir /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/database Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/.env Setup logs dir /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/logs Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/.env Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Setup database dir /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/database No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Setup logs dir /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/logs No config file found Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Setup directory for instance: node2 No config file found No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_concurrent_threads_soft_limit/configs/config_defined_50.xml'] to /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/database Setup logs dir /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers http://localhost:None "GET /version HTTP/1.1" 200 826 Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_concurrent_threads_soft_limit/configs/config_defined_1.xml'] to /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/configs/config.d Command:[docker compose --env-file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/.env --project-name roottestaccessforfunctions-gw7 --file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/docker-compose.yml pull] Setup database dir /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/database http://localhost:None "GET /version HTTP/1.1" 200 826 Setup logs dir /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node4 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/.env --project-name roottestbackuprestore-gw6 --file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/docker-compose.yml pull] Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_concurrent_threads_soft_limit/configs/config_limit_reached.xml'] to /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/configs/config.d Setup database dir /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/database http://localhost:None "GET /version HTTP/1.1" 200 826 Setup logs dir /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/.env Command:[docker compose --env-file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/.env --project-name roottestclusterallreplicas-gw1 --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/docker-compose.yml pull] Stdout:1 Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Volumes pruned: 1 http://localhost:None "GET /version HTTP/1.1" 200 826 Setup directory for instance: node1 Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Create directory for configuration generated in this helper Create directory for common tests configuration http://localhost:None "GET /version HTTP/1.1" 200 826 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/.env --project-name roottestasyncmetricsincgroup-gw5 --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/docker-compose.yml pull] Copy common configuration from helpers Generate and write macros file Command:[docker compose --env-file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/.env --project-name roottestchecktable-gw3 --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/docker-compose.yml pull] Command:[docker compose --env-file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/.env --project-name roottestaltercodec-gw9 --file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/docker-compose.yml pull] Copy custom test config files [] to /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/database Setup logs dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/configs/config.d Stdout:1 Stdout:1 Setup database dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/database Volumes pruned: 1 Setup directory for instance: server Volumes pruned: 1 Setup logs dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/logs Setup directory for instance: node1 Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/configs/config.d Generate and write macros file Generate and write macros file Setup database dir /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/database Setup logs dir /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/logs Copy custom test config files ['/ClickHouse/tests/integration/test_config_substitutions/configs/config_zk_include_test.xml'] to /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster3.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/replicated_access_storage.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/replicated_user_defined_sql_objects.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/backups_disk.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/lesser_timeouts.xml'] to /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/configs/config.d Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: clientA1 Create directory for configuration generated in this helper Setup database dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/database Create directory for common tests configuration Setup logs dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/logs Copy common configuration from helpers Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node4 Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/configs/config.d Create directory for configuration generated in this helper Setup database dir /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/database Create directory for common tests configuration Setup logs dir /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Copy common configuration from helpers Setup directory for instance: clientA2 Generate and write macros file Create directory for configuration generated in this helper Create directory for common tests configuration Copy custom test config files ['/ClickHouse/tests/integration/test_config_substitutions/configs/include_from_source.xml'] to /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/configs/config.d Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/database Setup database dir /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/database Setup logs dir /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: clientA3 http://localhost:None "GET /version HTTP/1.1" 200 826 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Command:[docker compose --env-file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/.env --project-name roottestconcurrentthreadssoftlimit-gw0 --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/docker-compose.yml pull] Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/database Setup logs dir /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/logs Setup database dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/database Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup logs dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/logs Setup logs dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/logs Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/.env Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] external_dir_abs_path=/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/backups Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Setup directory for instance: node5 Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Create directory for configuration generated in this helper Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Create directory for common tests configuration No config file found Copy common configuration from helpers Create directory for configuration generated in this helper Generate and write macros file Create directory for common tests configuration Copy custom test config files ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster3.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/replicated_access_storage.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/replicated_user_defined_sql_objects.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/backups_disk.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/lesser_timeouts.xml'] to /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/configs/config.d Copy common configuration from helpers Generate and write macros file Setup database dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/database Setup logs dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" external_dir_abs_path=/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/backups Copy custom test config files [] to /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/configs/config.d Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for common tests configuration Setup database dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/database Copy common configuration from helpers Setup logs dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Generate and write macros file Setup directory for instance: node6 Copy custom test config files ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster3.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/replicated_access_storage.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/replicated_user_defined_sql_objects.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/backups_disk.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/lesser_timeouts.xml'] to /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/configs/config.d Create directory for configuration generated in this helper Create directory for common tests configuration Setup database dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/database Setup logs dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] external_dir_abs_path=/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/backups Copy common configuration from helpers Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/.env Generate and write macros file Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Copy custom test config files ['/ClickHouse/tests/integration/test_config_substitutions/configs/include_from_source.xml'] to /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/configs/config.d Setup database dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/database Setup logs dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node7 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_config_substitutions/configs/000-server_overrides.xml', '/ClickHouse/tests/integration/test_config_substitutions/configs/010-server_with_env_subst.xml'] to /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/configs/config.d Setup database dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/database Setup logs dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MAX_QUERY_SIZE': '121212', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/coordination', 'INCLUDE_FROM_ENV': '/etc/clickhouse-server/config.d/include_from_source.xml', 'MAX_THREADS': '2'} stored in /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/.env http://localhost:None "GET /version HTTP/1.1" 200 826 Setup directory for instance: node8 Command:[docker compose --env-file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/.env --project-name roottestallowedclienthosts-gw8 --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/docker-compose.yml pull] Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_config_substitutions/configs/include_from_source.yml'] to /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/configs/config.d Setup database dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/database Setup logs dir /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MAX_QUERY_SIZE': '55555', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/coordination', 'INCLUDE_FROM_ENV': '/etc/clickhouse-server/config.d/include_from_source.xml'} stored in /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/.env --project-name roottestconfigsubstitutions-gw4 --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/docker-compose.yml pull] http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/.env --project-name roottestbackuprestoreoncluster-gw2 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/docker-compose.yml pull] Stderr: node1 Pulling Stderr: node1 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/.env --project-name roottestaltercodec-gw9 --file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/.env --project-name roottestaltercodec-gw9 --file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/docker-compose.yml up -d --no-recreate] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/.env --project-name roottestbackuprestore-gw6 --file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/.env --project-name roottestbackuprestore-gw6 --file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/docker-compose.yml up -d --no-recreate] Stderr: node2 Skipped - Image is already being pulled by zoo3 Stderr: node3 Skipped - Image is already being pulled by zoo3 Stderr: node1 Skipped - Image is already being pulled by zoo3 Stderr: zoo1 Skipped - Image is already being pulled by zoo3 Stderr: zoo2 Skipped - Image is already being pulled by zoo3 Stderr: zoo3 Pulling Stderr: zoo3 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper1/log', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper1/config', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper1/coordination', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper2/log', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper2/config', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper2/coordination', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper3/log', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper3/config', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/keeper3/coordination'] Command:[docker compose --project-name roottestbackuprestoreoncluster-gw2 --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: clientA3 Skipped - Image is already being pulled by clientA2 Stderr: server Skipped - Image is already being pulled by clientA2 Stderr: clientA1 Skipped - Image is already being pulled by clientA2 Stderr: clientA2 Pulling Stderr: clientA2 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/.env --project-name roottestallowedclienthosts-gw8 --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/.env --project-name roottestallowedclienthosts-gw8 --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/docker-compose.yml up -d --no-recreate] Stderr: node2 Skipped - Image is already being pulled by zoo1 Stderr: node6 Skipped - Image is already being pulled by zoo1 Stderr: node3 Skipped - Image is already being pulled by zoo1 Stderr: node4 Skipped - Image is already being pulled by zoo1 Stderr: node7 Skipped - Image is already being pulled by zoo1 Stderr: node8 Skipped - Image is already being pulled by zoo1 Stderr: node1 Skipped - Image is already being pulled by zoo1 Stderr: node5 Skipped - Image is already being pulled by zoo1 Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: zoo1 Pulling Stderr: zoo1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/log', '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/config', '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper1/coordination', '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/log', '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/config', '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper2/coordination', '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/log', '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/config', '/ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/keeper3/coordination'] Command:[docker compose --project-name roottestconfigsubstitutions-gw4 --env-file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: node3 Skipped - Image is already being pulled by node2 Stderr: node4 Skipped - Image is already being pulled by node2 Stderr: node1 Skipped - Image is already being pulled by node2 Stderr: node2 Pulling Stderr: node2 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/.env --project-name roottestconcurrentthreadssoftlimit-gw0 --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/.env --project-name roottestconcurrentthreadssoftlimit-gw0 --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/docker-compose.yml up -d --no-recreate] Stderr: instance Pulling Stderr: instance Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/.env --project-name roottestaccessforfunctions-gw7 --file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/.env --project-name roottestaccessforfunctions-gw7 --file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/docker-compose.yml up -d --no-recreate] Stderr: node2 Skipped - Image is already being pulled by node1 Stderr: node1 Pulling Stderr: node1 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/.env --project-name roottestasyncmetricsincgroup-gw5 --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/.env --project-name roottestasyncmetricsincgroup-gw5 --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/docker-compose.yml up -d --no-recreate] Stderr: Network roottestaltercodec-gw9_default Creating Stderr: Network roottestaltercodec-gw9_default Created Stderr: Container roottestaltercodec-gw9-node1-1 Creating Stderr: Container roottestaltercodec-gw9-node1-1 Created Stderr: Container roottestaltercodec-gw9-node1-1 Starting Stderr: Container roottestaltercodec-gw9-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestaltercodec-gw9-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestaltercodec-gw9-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.2... http://localhost:None "GET /v1.46/containers/roottestaltercodec-gw9-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None Stderr: node1 Skipped - Image is already being pulled by node2 Stderr: node2 Pulling Stderr: node2 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/.env --project-name roottestclusterallreplicas-gw1 --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/.env --project-name roottestclusterallreplicas-gw1 --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/docker-compose.yml up -d --no-recreate] http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None Stderr: Network roottestbackuprestore-gw6_default Creating Stderr: Network roottestbackuprestore-gw6_default Created Stderr: Container roottestbackuprestore-gw6-node-1 Creating Stderr: Container roottestbackuprestore-gw6-node-1 Created Stderr: Container roottestbackuprestore-gw6-node-1 Starting Stderr: Container roottestbackuprestore-gw6-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestbackuprestore-gw6-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestbackuprestore-gw6-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.2.2... http://localhost:None "GET /v1.46/containers/roottestbackuprestore-gw6-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None Stderr: node1 Skipped - Image is already being pulled by node2 Stderr: zoo1 Skipped - Image is already being pulled by node2 Stderr: zoo2 Skipped - Image is already being pulled by node2 Stderr: zoo3 Skipped - Image is already being pulled by node2 Stderr: node2 Pulling Stderr: node2 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper1/log', '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper1/config', '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper1/coordination', '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper2/log', '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper2/config', '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper2/coordination', '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper3/log', '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper3/config', '/ClickHouse/tests/integration/test_check_table/_instances-0-gw3/keeper3/coordination'] Command:[docker compose --project-name roottestchecktable-gw3 --env-file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None Stderr:time="2025-04-02T02:52:34Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestbackuprestoreoncluster-gw2_default Creating Stderr: Network roottestbackuprestoreoncluster-gw2_default Created Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Creating Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Creating Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Creating Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Created Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Created Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Created Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Starting Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Starting Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Starting Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Started Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Started Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Started Stderr:time="2025-04-02T02:52:34Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:52:34Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.3.3, port:2181, use_ssl:False Connecting to 172.16.3.3(172.16.3.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None Connecting to 172.16.3.3(172.16.3.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None Connecting to 172.16.3.3(172.16.3.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None Stderr:time="2025-04-02T02:52:34Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestconfigsubstitutions-gw4_default Creating Stderr: Network roottestconfigsubstitutions-gw4_default Created Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Created Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Created Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Created Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Started Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Started Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Started Stderr:time="2025-04-02T02:52:35Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:52:35Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.4.2, port:2181, use_ssl:False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/43192f80ecc9dcc6b509a0009ec4ee3661abdc2cb83b03876de073383ae021a8/json HTTP/1.1" 200 None ClickHouse node1 started Executing query CREATE TABLE test_alter_codec_index (`id` UInt64, value UInt64, INDEX id_index id TYPE minmax GRANULARITY 1) Engine=MergeTree() ORDER BY tuple() on node1 http://localhost:None "GET /v1.46/containers/51df6a5dcdcfb7a50a91174a99a9f22f0eb7cae87e44221ddf87f2ad605fe7f9/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE DATABASE test ENGINE = Ordinary on node Stderr: Network roottestallowedclienthosts-gw8_default Creating Stderr: Network roottestallowedclienthosts-gw8_default Created Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Creating Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Creating Stderr: Container roottestallowedclienthosts-gw8-server-1 Creating Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Creating Stderr: Container roottestallowedclienthosts-gw8-server-1 Created Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Created Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Created Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Created Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Starting Stderr: Container roottestallowedclienthosts-gw8-server-1 Starting Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Starting Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Starting Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Started Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Started Stderr: Container roottestallowedclienthosts-gw8-server-1 Started Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Started Connecting to 172.16.3.3(172.16.3.3):2181, use_ssl: False ClickHouse instance created get_instance_ip instance_name=server Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-server-1/json HTTP/1.1" 200 None get_instance_ip instance_name=server http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-server-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in server, ip: 172.16.5.5... http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-server-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None Stderr: Network roottestconcurrentthreadssoftlimit-gw0_default Creating Stderr: Network roottestconcurrentthreadssoftlimit-gw0_default Created Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Creating Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Creating Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Creating Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Creating Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Created Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Created Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Created Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Created Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Starting Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Starting Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Starting Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Starting Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Started Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Started Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Started Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.6.4... http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None Stderr: Network roottestaccessforfunctions-gw7_default Creating Stderr: Network roottestaccessforfunctions-gw7_default Created Stderr: Container roottestaccessforfunctions-gw7-instance-1 Creating Stderr: Container roottestaccessforfunctions-gw7-instance-1 Created Stderr: Container roottestaccessforfunctions-gw7-instance-1 Starting Stderr: Container roottestaccessforfunctions-gw7-instance-1 Started ClickHouse instance created get_instance_ip instance_name=instance http://localhost:None "GET /v1.46/containers/roottestaccessforfunctions-gw7-instance-1/json HTTP/1.1" 200 None get_instance_ip instance_name=instance http://localhost:None "GET /v1.46/containers/roottestaccessforfunctions-gw7-instance-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in instance, ip: 172.16.7.2... http://localhost:None "GET /v1.46/containers/roottestaccessforfunctions-gw7-instance-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS test.tbl on node Executing query INSERT INTO test_alter_codec_index SELECT number, number * number from numbers(100) on node1 http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None Stderr: Network roottestasyncmetricsincgroup-gw5_default Creating Stderr: Network roottestasyncmetricsincgroup-gw5_default Created Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Creating Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Creating Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Created Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Created Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Starting Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Starting Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Started Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestasyncmetricsincgroup-gw5-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestasyncmetricsincgroup-gw5-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.8.2... http://localhost:None "GET /v1.46/containers/roottestasyncmetricsincgroup-gw5-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None Executing query CREATE TABLE test.tbl (p Date, k Int8) ENGINE = MergeTree PARTITION BY toYYYYMM(p) ORDER BY p on node http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None Executing query ALTER TABLE test_alter_codec_index MODIFY COLUMN id UInt64 CODEC(NONE) on node1 http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None Stderr: Network roottestclusterallreplicas-gw1_default Creating Stderr: Network roottestclusterallreplicas-gw1_default Created Stderr: Container roottestclusterallreplicas-gw1-node2-1 Creating Stderr: Container roottestclusterallreplicas-gw1-node1-1 Creating Stderr: Container roottestclusterallreplicas-gw1-node1-1 Created Stderr: Container roottestclusterallreplicas-gw1-node2-1 Created Stderr: Container roottestclusterallreplicas-gw1-node1-1 Starting Stderr: Container roottestclusterallreplicas-gw1-node2-1 Starting Stderr: Container roottestclusterallreplicas-gw1-node2-1 Started Stderr: Container roottestclusterallreplicas-gw1-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestclusterallreplicas-gw1-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestclusterallreplicas-gw1-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.9.3... http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestclusterallreplicas-gw1-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None Executing query INSERT INTO test.tbl (p, k) VALUES(toDate(1), 1) on node Executing query ALTER TABLE test_alter_codec_index MODIFY COLUMN id UInt64 CODEC(Delta, LZ4) on node1 http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None Executing query INSERT INTO test.tbl (p, k) VALUES(toDate(2), 2) on node http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None Executing query ALTER TABLE test_alter_codec_index MODIFY COLUMN id UInt32 CODEC(Delta, LZ4) on node1 http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None Stderr:time="2025-04-02T02:52:34Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestchecktable-gw3_default Creating Stderr: Network roottestchecktable-gw3_default Created Stderr: Container roottestchecktable-gw3-zoo3-1 Creating Stderr: Container roottestchecktable-gw3-zoo1-1 Creating Stderr: Container roottestchecktable-gw3-zoo2-1 Creating Stderr: Container roottestchecktable-gw3-zoo1-1 Created Stderr: Container roottestchecktable-gw3-zoo2-1 Created Stderr: Container roottestchecktable-gw3-zoo3-1 Created Stderr: Container roottestchecktable-gw3-zoo1-1 Starting Stderr: Container roottestchecktable-gw3-zoo2-1 Starting Stderr: Container roottestchecktable-gw3-zoo3-1 Starting Stderr: Container roottestchecktable-gw3-zoo2-1 Started Stderr: Container roottestchecktable-gw3-zoo1-1 Started Stderr: Container roottestchecktable-gw3-zoo3-1 Started Stderr:time="2025-04-02T02:52:36Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:52:36Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.10.4, port:2181, use_ssl:False Connecting to 172.16.10.4(172.16.10.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None Connecting to 172.16.10.4(172.16.10.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query INSERT INTO test.tbl (p, k) VALUES(toDate(3), 3) on node http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None Executing query ALTER TABLE test_alter_codec_index MODIFY COLUMN id UInt64 DEFAULT 3 CODEC(Delta, LZ4) on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None Connecting to 172.16.10.4(172.16.10.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None Executing query INSERT INTO test.tbl (p, k) VALUES(toDate(31), 31) on node http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None Executing query INSERT INTO test_alter_codec_index (value) VALUES (1) on node1 http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f132b5709d07f8a6551599b9d617ee7517b0fb93510aea2328aee23ebd72ed8e/json HTTP/1.1" 200 None ClickHouse server started get_instance_ip instance_name=clientA1 http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=clientA1 http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in clientA1, ip: 172.16.5.3... http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/325246ad6a3246f532835c0a3d625690faf038ea1d1968ef35d81bc596b6b364/json HTTP/1.1" 200 None ClickHouse clientA1 started get_instance_ip instance_name=clientA2 http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=clientA2 http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in clientA2, ip: 172.16.5.4... http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/79e1b8d3bb94e3cae9d70a715fbc192ff6bc7b20f33d9e36f72af1827a771bcf/json HTTP/1.1" 200 None ClickHouse clientA2 started get_instance_ip instance_name=clientA3 http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=clientA3 http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in clientA3, ip: 172.16.5.2... http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestallowedclienthosts-gw8-clientA3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/21b9c936aa23925095f40182f2917e2283811ee3b9ef52bf4b8aa3abbb271aca/json HTTP/1.1" 200 None Connecting to 172.16.10.4(172.16.10.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query INSERT INTO test.tbl (p, k) VALUES(toDate(32), 32) on node http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT sum(id) FROM test_alter_codec_index on node1 http://localhost:None "GET /v1.46/containers/74f9d70ed870730b91ba8cca94a18a760312cdfcdcc4b49bcdcbf7430dab21ed/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.6.5... http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ae7cd1e6636b818e58ea9ef08e59813845e20f7e8798c80e82cdd0ca12993559/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/21b9c936aa23925095f40182f2917e2283811ee3b9ef52bf4b8aa3abbb271aca/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.6.3... ClickHouse clientA3 started run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s server:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-server-1 bash -c /usr/bin/curl -s server:8123 ] http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/599a17f263ef833dddc40a874085a9377268cce7b8376a040cbebdfabcb11621/json HTTP/1.1" 200 None ClickHouse node3 started get_instance_ip instance_name=node4 http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node4-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node4 http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node4-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node4, ip: 172.16.6.2... http://localhost:None "GET /v1.46/containers/roottestconcurrentthreadssoftlimit-gw0-node4-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/526cf5c36ec6d26308a8d40ae2333b87a860572ffd4e0080955e5f30017f496b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/52cd55deade94357c7ddd88fa91565cd94844680907f5c167a666a814557330e/json HTTP/1.1" 200 None ClickHouse instance started Executing query CREATE USER A on instance http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 ip address show] http://localhost:None "GET /v1.46/containers/0cd3c1b1bfcf30a8b39732a8ffd044845a4e532f231ff4933825afe01519ec77/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestasyncmetricsincgroup-gw5-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestasyncmetricsincgroup-gw5-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.8.3... http://localhost:None "GET /v1.46/containers/roottestasyncmetricsincgroup-gw5-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/71b510ae9d33090f6dee3f314a906c9dfb8f204af66e71649a7bb07f99c277ed/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/526cf5c36ec6d26308a8d40ae2333b87a860572ffd4e0080955e5f30017f496b/json HTTP/1.1" 200 None Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:33: eth0@if34: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.5/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA1.com] Executing query INSERT INTO test.tbl (p, k) VALUES(toDate(33), 33) on node http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/71b510ae9d33090f6dee3f314a906c9dfb8f204af66e71649a7bb07f99c277ed/json HTTP/1.1" 200 None Executing query CREATE USER B on instance Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA2.com] http://localhost:None "GET /v1.46/containers/526cf5c36ec6d26308a8d40ae2333b87a860572ffd4e0080955e5f30017f496b/json HTTP/1.1" 200 None ClickHouse node4 started Executing query ALTER TABLE test_alter_codec_index MODIFY COLUMN id UInt64 ALIAS 3 CODEC(Delta, LZ4) on node1 Executing query SELECT count(*) FROM numbers_mt(10000000) on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA3.com] http://localhost:None "GET /v1.46/containers/71b510ae9d33090f6dee3f314a906c9dfb8f204af66e71649a7bb07f99c277ed/json HTTP/1.1" 200 None ClickHouse node2 started Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on node1 Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 33: eth0@if34: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.5/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host server --query 'DROP TABLE IF EXISTS test_allowed_client_hosts'"] Command:[docker exec roottestallowedclienthosts-gw8-server-1 bash -c /usr/bin/clickhouse client --host server --query 'DROP TABLE IF EXISTS test_allowed_client_hosts'] Executing query SELECT * FROM test.tbl ORDER BY p on node http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query CREATE FUNCTION MySum AS (a, b) -> a + b on instance Connecting to 172.16.3.3(172.16.3.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query ALTER TABLE test_alter_codec_index MODIFY COLUMN id UInt64 MATERIALIZED 3 CODEC(Delta, LZ4) on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None [gw5] SKIPPED test_async_metrics_in_cgroup/test.py::test_normalized_user_cpu test_async_metrics_in_cgroup/test.py::test_system_wide_metrics Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on node1 Executing query SYSTEM FLUSH LOGS on node1 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s server:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-server-1 bash -c /usr/bin/curl -s server:8123 ] Connecting to 172.16.10.4(172.16.10.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query GRANT CREATE FUNCTION on *.* TO A on instance Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 ip address show] Executing query ALTER TABLE test.tbl FREEZE on node Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:33: eth0@if34: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.5/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA1.com] Executing query INSERT INTO test_alter_codec_index (value) VALUES (1) on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA2.com] [gw5] SKIPPED test_async_metrics_in_cgroup/test.py::test_system_wide_metrics Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on node1 test_async_metrics_in_cgroup/test.py::test_user_cpu_accounting Executing query CREATE FUNCTION MySum AS (a, b) -> a + b on instance Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA3.com] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query CREATE TABLE test.tbl2 AS test.tbl on node Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 33: eth0@if34: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.5/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host server --query 'CREATE TABLE test_allowed_client_hosts (x Int32) ENGINE = MergeTree() ORDER BY tuple()'"] Command:[docker exec roottestallowedclienthosts-gw8-server-1 bash -c /usr/bin/clickhouse client --host server --query 'CREATE TABLE test_allowed_client_hosts (x Int32) ENGINE = MergeTree() ORDER BY tuple()'] Executing query SELECT sum(id) FROM test_alter_codec_index on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT MySum(1, 2) on instance Command:[docker compose --env-file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/.env --project-name roottestasyncmetricsincgroup-gw5 --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/docker-compose.yml stop --timeout 20] [gw5] SKIPPED test_async_metrics_in_cgroup/test.py::test_user_cpu_accounting http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query INSERT INTO test.tbl2(p, k) VALUES(toDate(3), 3) on node run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s server:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-server-1 bash -c /usr/bin/curl -s server:8123 ] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 ip address show] Executing query DROP FUNCTION MySum on instance Executing query ALTER TABLE test_alter_codec_index MODIFY COLUMN id UInt64 on node1 Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:33: eth0@if34: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.5/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA1.com] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA2.com] Executing query INSERT INTO test.tbl2(p, k) VALUES(toDate(4), 4) on node Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA3.com] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 33: eth0@if34: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.5/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host server --query 'INSERT INTO test_allowed_client_hosts VALUES (5)'"] Command:[docker exec roottestallowedclienthosts-gw8-server-1 bash -c /usr/bin/clickhouse client --host server --query 'INSERT INTO test_allowed_client_hosts VALUES (5)'] Executing query select ProfileEvents['ConcurrencyControlSlotsGranted'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_1' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query ALTER TABLE test_alter_codec_index MODIFY COLUMN id Int64 on node1 Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Stopping Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Stopping Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Stopped Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/.env --project-name roottestasyncmetricsincgroup-gw5 --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_async_metrics_in_cgroup/_instances-0-gw5/node2/docker-compose.yml down --volumes] Executing query INSERT INTO test.tbl2(p, k) VALUES(toDate(33), 33) on node Executing query GRANT DROP FUNCTION ON *.* TO B on instance http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.4.3, port:2181, use_ssl:False Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Connecting to 172.16.10.4(172.16.10.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s server:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-server-1 bash -c /usr/bin/curl -s server:8123 ] Executing query select ProfileEvents['ConcurrencyControlSlotsDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_1' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.4.4, port:2181, use_ssl:False Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 ip address show] Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') get_instance_ip instance_name=zoo1 Executing query DROP TABLE IF EXISTS test_alter_codec_index on node1 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.4.2, port:2181, use_ssl:False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED run_kazoo_commands_with_retries: zoo1, .create_zk_roots at 0x7fdf985ce170> Sending request(xid=1): Create(path='/setting/max_query_size', data=b'77777', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) Received error(xid=1) NoNodeError() Sending request(xid=2): Exists(path='/setting', watcher=None) Sending request(xid=3): Exists(path='/', watcher=None) Received response(xid=3): ZnodeStat(czxid=0, mzxid=0, ctime=0, mtime=0, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=1, pzxid=0) Sending request(xid=4): Create(path='/setting', data=b'', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) Received response(xid=4): '/setting' Sending request(xid=5): Create(path='/setting/max_query_size', data=b'77777', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) Received response(xid=5): '/setting/max_query_size' Sending request(xid=6): Create(path='/users_from_zk_1', data=b'default', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) Received response(xid=6): '/users_from_zk_1' Sending request(xid=7): Create(path='/users_from_zk_2', data=b'default', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) Received response(xid=7): '/users_from_zk_2' Sending request(xid=8): Create(path='/min_bytes_for_wide_part', data=b'33', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Received response(xid=8): '/min_bytes_for_wide_part' Sending request(xid=9): Create(path='/merge_max_block_size', data=b'8888', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) Received response(xid=9): '/merge_max_block_size' ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/.env --project-name roottestconfigsubstitutions-gw4 --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/.env --project-name roottestconfigsubstitutions-gw4 --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/docker-compose.yml up -d --no-recreate] Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:33: eth0@if34: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.5/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA1.com] Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA2.com] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None [gw9] PASSED test_alter_codec/test.py::test_alter_codec_index test_alter_codec/test.py::test_alter_codec_pk Executing query CREATE TABLE test_alter_codec_pk (id UInt64, value UInt64) Engine=MergeTree() ORDER BY id on node1 Executing query DROP FUNCTION MySum on instance Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-server-1 host clientA3.com] Executing query select ProfileEvents['ConcurrencyControlSlotsAcquired'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_1' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 Executing query INSERT INTO test.tbl2(p, k) VALUES(toDate(34), 34) on node http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 33: eth0@if34: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.5/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-server-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host server --query 'SELECT fqdn(), hostName()'"] Command:[docker exec roottestallowedclienthosts-gw8-server-1 bash -c /usr/bin/clickhouse client --host server --query 'SELECT fqdn(), hostName()'] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query INSERT INTO test_alter_codec_pk SELECT number, number * number from numbers(100) on node1 Executing query SELECT MySum(1, 2) on instance Executing query select ProfileEvents['ConcurrencyControlSlotsAcquiredNonCompeting'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_1' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Stopping Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Stopping Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Stopped Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Removing Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Stopped Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Removing Stderr: Container roottestasyncmetricsincgroup-gw5-node1-1 Removed Stderr: Container roottestasyncmetricsincgroup-gw5-node2-1 Removed Stderr: Network roottestasyncmetricsincgroup-gw5_default Removing Stderr: Network roottestasyncmetricsincgroup-gw5_default Removed Cleanup called Docker networks for project roottestasyncmetricsincgroup-gw5 are NETWORK ID NAME DRIVER SCOPE Stdout:server server run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s clientA1.com:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 bash -c /usr/bin/curl -s clientA1.com:8123 ] Executing query SELECT * FROM test.tbl2 ORDER BY p on node Docker containers for project roottestasyncmetricsincgroup-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestasyncmetricsincgroup-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestasyncmetricsincgroup-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestasyncmetricsincgroup-gw5 Trying to prune unused networks... Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 ip address show] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_authentication/test.py::test_authentication_fail Running tests in /ClickHouse/tests/integration/test_authentication/test.py Cluster start called. is_up=False Docker networks for project roottestauthentication-gw5 are NETWORK ID NAME DRIVER SCOPE Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:29: eth0@if30: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.3/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 host clientA1.com] Docker containers for project roottestauthentication-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestauthentication-gw5 are DRIVER VOLUME NAME Cleanup called http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Docker networks for project roottestauthentication-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestauthentication-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 host clientA2.com] Docker volumes for project roottestauthentication-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestauthentication-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestauthentication-gw5 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query ALTER TABLE test_alter_codec_pk MODIFY COLUMN id UInt64 CODEC(NONE) on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 host clientA3.com] Executing query REVOKE CREATE FUNCTION ON *.* FROM A on instance Stdout:1 Volumes pruned: 1 Setup directory for instance: instance Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/configs/config.d Setup database dir /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/database Setup logs dir /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/.env --project-name roottestauthentication-gw5 --file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/docker-compose.yml pull] Executing query select ProfileEvents['ConcurrencyControlQueriesDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_1' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 29: eth0@if30: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.3/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host clientA1.com --query 'SELECT fqdn(), hostName()'"] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 bash -c /usr/bin/clickhouse client --host clientA1.com --query 'SELECT fqdn(), hostName()'] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Running Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Running Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Running Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Creating Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Created Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Created Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Created Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Created Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Created Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Created Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Created Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Created Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Starting Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Started Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Started Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Started Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Started Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Started Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Started Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Started Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.4.10... http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Executing query ALTER TABLE test_alter_codec_pk MODIFY COLUMN id UInt64 CODEC(Delta, LZ4) on node1 Stdout:clientA1.com clientA1.com run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s clientA2.com:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 bash -c /usr/bin/curl -s clientA2.com:8123 ] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query CREATE FUNCTION MySum AS (a, b) -> a + b on instance Executing query select length(thread_ids) from system.query_log where current_database = currentDatabase() and type = 'QueryFinish' and query_id = 'test_concurrent_threads_soft_limit_1' order by query_start_time_microseconds desc limit 1 on node1 Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 ip address show] copy from /var/lib/clickhouse/shadow/*/data/test/tbl to /var/lib/clickhouse/data/test/tbl2/detached run container_id:roottestbackuprestore-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cp -r /var/lib/clickhouse/shadow/*/data/test/tbl -T /var/lib/clickhouse/data/test/tbl2/detached'] Command:[docker exec roottestbackuprestore-gw6-node-1 bash -c cp -r /var/lib/clickhouse/shadow/*/data/test/tbl -T /var/lib/clickhouse/data/test/tbl2/detached] http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:31: eth0@if32: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.4/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 host clientA1.com] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query ALTER TABLE test.tbl2 ATTACH PARTITION 197001 on node Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 host clientA2.com] http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Executing query SELECT sum(id) FROM test_alter_codec_pk on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 host clientA3.com] Executing query DROP USER IF EXISTS A on instance [gw0] PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default Executing query SELECT count(*) FROM numbers_mt(10000000) on node3 test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_1 http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 31: eth0@if32: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.4/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host clientA2.com --query 'SELECT fqdn(), hostName()'"] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 bash -c /usr/bin/clickhouse client --host clientA2.com --query 'SELECT fqdn(), hostName()'] Executing query ALTER TABLE test.tbl2 ATTACH PARTITION 197002 on node http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Executing query DROP USER IF EXISTS B on instance http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query ALTER TABLE test_alter_codec_pk MODIFY COLUMN id UInt32 CODEC(Delta, LZ4) on node1 Executing query SELECT sleep(2) on node http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Stdout:clientA2.com clientA2.com run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s clientA3.com:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 bash -c /usr/bin/curl -s clientA3.com:8123 ] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SYSTEM FLUSH LOGS on node3 run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestaccessforfunctions-gw7-instance-1 bash -c ps -C clickhouse] [gw7] PASSED test_access_for_functions/test.py::test_access_rights_for_function test_access_for_functions/test.py::test_ignore_obsolete_grant_on_database Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 ip address show] http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Stdout: PID TTY TIME CMD Stdout: 8 ? 00:00:01 clickhouse run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestaccessforfunctions-gw7-instance-1 bash -c pkill clickhouse] Connecting to 172.16.3.3(172.16.3.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:26: eth0@if27: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Zookeeper connection established, state: CONNECTED Stdout: inet 172.16.5.2/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 host clientA1.com] Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query ALTER TABLE test_alter_codec_pk MODIFY COLUMN id UInt64 DEFAULT 3 CODEC(Delta, LZ4) on node1 run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 host clientA2.com] Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.3.4, port:2181, use_ssl:False Connecting to 172.16.3.4(172.16.3.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 host clientA3.com] Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.3.2, port:2181, use_ssl:False Connecting to 172.16.3.2(172.16.3.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 26: eth0@if27: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.2/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 Executing query INSERT INTO test_alter_codec_pk (value) VALUES (1) on node1 run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host clientA3.com --query 'SELECT fqdn(), hostName()'"] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 bash -c /usr/bin/clickhouse client --host clientA3.com --query 'SELECT fqdn(), hostName()'] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Connecting to 172.16.10.4(172.16.10.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/.env --project-name roottestbackuprestoreoncluster-gw2 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/.env --project-name roottestbackuprestoreoncluster-gw2 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/docker-compose.yml up -d --no-recreate] http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.10.3, port:2181, use_ssl:False Connecting to 172.16.10.3(172.16.10.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Executing query SELECT sum(id) FROM test_alter_codec_pk on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Stdout:clientA3.com clientA3.com s:server server , a1:clientA1.com clientA1.com , a2:clientA2.com clientA2.com , a3:clientA3.com clientA3.com run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s server:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 bash -c /usr/bin/curl -s server:8123 ] Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 ip address show] http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.10.2, port:2181, use_ssl:False Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:29: eth0@if30: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.3/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 host clientA1.com] Connecting to 172.16.10.2(172.16.10.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 host clientA2.com] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/.env --project-name roottestchecktable-gw3 --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/.env --project-name roottestchecktable-gw3 --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/docker-compose.yml up -d --no-recreate] Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 host clientA3.com] Executing query ALTER TABLE test_alter_codec_pk MODIFY COLUMN id UInt64 ALIAS 3 CODEC(Delta, LZ4) on node1 http://localhost:None "GET /v1.46/containers/ed83d509978667c665afb645789b66310b0e2656165b385fa87ca1a3124f26cd/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.4.12... Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 29: eth0@if30: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.3/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-clientA1-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host server --query 'SELECT * FROM test_allowed_client_hosts'"] Command:[docker exec roottestallowedclienthosts-gw8-clientA1-1 bash -c /usr/bin/clickhouse client --host server --query 'SELECT * FROM test_allowed_client_hosts'] http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7de4c76cfb7b00c2f766713e6f8806ab9cf57fc34d47a5cce8c257b5c4185429/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.4.9... http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b0ecd955bb33495e5845b00e51c122c4c0c5b91b603db504c12c4164733e8354/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b0ecd955bb33495e5845b00e51c122c4c0c5b91b603db504c12c4164733e8354/json HTTP/1.1" 200 None ClickHouse node3 started get_instance_ip instance_name=node4 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node4-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node4 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node4-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node4, ip: 172.16.4.6... http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node4-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e261fdc4d054b1fd68e91f1c6231b8698d2080b77eb7d33dbf7ff4b2a699dfca/json HTTP/1.1" 200 None ClickHouse node4 started get_instance_ip instance_name=node5 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node5-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node5 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node5-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node5, ip: 172.16.4.7... http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node5-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c1434de000cd7d87631c1787f70fbd1a4275d46cb0359be7c032806ac43ebff3/json HTTP/1.1" 200 None ClickHouse node5 started get_instance_ip instance_name=node6 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node6-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node6 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node6-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node6, ip: 172.16.4.11... http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node6-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/54469e2d8765ed321c7646b03e6f7bb501159bebf22f8a4c13e927971c0302ff/json HTTP/1.1" 200 None ClickHouse node6 started get_instance_ip instance_name=node7 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node7-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node7 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node7-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node7, ip: 172.16.4.5... http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node7-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/3b431c2afb5b2a60780c16ca326c09ef70c19a46e590e6b27d8a3d726c9d8c35/json HTTP/1.1" 200 None ClickHouse node7 started get_instance_ip instance_name=node8 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node8-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node8 http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node8-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node8, ip: 172.16.4.8... http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-node8-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/51ac66142356be6000edeeb8bdec3325815bfbe3959ceb602646a015e087f290/json HTTP/1.1" 200 None ClickHouse node8 started Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Running Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Running Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Running Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Creating Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Creating Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Creating Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Created Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Created Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Created Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Starting Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Starting Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Starting Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Started Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Started Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Started Executing query DROP DATABASE IF EXISTS db1 SYNC on node5 ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.3.6... http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlSlotsGranted'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_3' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node3 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:5 run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s server:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 bash -c /usr/bin/curl -s server:8123 ] http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 ip address show] Executing query ALTER TABLE test_alter_codec_pk MODIFY COLUMN id UInt64 MATERIALIZED 3 CODEC(Delta, LZ4) on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:26: eth0@if27: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.2/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 host clientA1.com] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:8 Stderr: Container roottestchecktable-gw3-zoo1-1 Running Stderr: Container roottestchecktable-gw3-zoo3-1 Running Stderr: Container roottestchecktable-gw3-zoo2-1 Running Stderr: Container roottestchecktable-gw3-node1-1 Creating Stderr: Container roottestchecktable-gw3-node2-1 Creating Stderr: Container roottestchecktable-gw3-node2-1 Created Stderr: Container roottestchecktable-gw3-node1-1 Created Stderr: Container roottestchecktable-gw3-node2-1 Starting Stderr: Container roottestchecktable-gw3-node1-1 Starting Stderr: Container roottestchecktable-gw3-node2-1 Started Stderr: Container roottestchecktable-gw3-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 host clientA2.com] http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.10.6... http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Executing query CREATE DATABASE db1 on node5 Executing query INSERT INTO test_alter_codec_pk (value) VALUES (1) on node1 Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 host clientA3.com] Executing query select ProfileEvents['ConcurrencyControlSlotsDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_3' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node3 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 26: eth0@if27: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.2/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-clientA3-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host server --query 'SELECT * FROM test_allowed_client_hosts'"] Command:[docker exec roottestallowedclienthosts-gw8-clientA3-1 bash -c /usr/bin/clickhouse client --host server --query 'SELECT * FROM test_allowed_client_hosts'] http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Executing query CREATE TABLE db1.test_table(date Date, k1 String, v1 Int32) ENGINE = MergeTree(date, (k1, date), 8192) on node5 Executing query SELECT sum(id) FROM test_alter_codec_pk on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlSlotsAcquired'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_3' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node3 http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Stdout:5 [gw8] PASSED test_allowed_client_hosts/test.py::test_allowed_host test_allowed_client_hosts/test.py::test_denied_host run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['bash', '-c', '/usr/bin/curl -s server:8123 '] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 bash -c /usr/bin/curl -s server:8123 ] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:Ok. run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['ip', 'address', 'show'] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 ip address show] http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Executing query INSERT INTO db1.test_table VALUES('2000-01-01', 'test_key', 1) on node5 Stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 Stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 Stdout: inet 127.0.0.1/8 scope host lo Stdout: valid_lft forever preferred_lft forever Stdout: inet6 ::1/128 scope host Stdout: valid_lft forever preferred_lft forever Stdout:31: eth0@if32: mtu 1500 qdisc noqueue state UP group default Stdout: link/ether 02:42:ac:10:05:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0 Stdout: inet 172.16.5.4/24 brd 172.16.5.255 scope global eth0 Stdout: valid_lft forever preferred_lft forever run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['host', 'clientA1.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 host clientA1.com] Executing query ALTER TABLE test_alter_codec_pk MODIFY COLUMN id UInt64 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlSlotsAcquiredNonCompeting'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_3' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node3 Stdout:clientA1.com has address 172.16.5.3 run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['host', 'clientA2.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 host clientA2.com] http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stdout:clientA2.com has address 172.16.5.4 run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['host', 'clientA3.com'] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 host clientA3.com] http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None Executing query ALTER TABLE test_alter_codec_pk MODIFY COLUMN id Int64 on node1 http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Stdout:clientA3.com has address 172.16.5.2 IP: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 31: eth0@if32: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:10:05:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.16.5.4/24 brd 172.16.5.255 scope global eth0 valid_lft forever preferred_lft forever , A1 clientA1.com has address 172.16.5.3 , A2 clientA2.com has address 172.16.5.4 , A3 clientA3.com has address 172.16.5.2 run container_id:roottestallowedclienthosts-gw8-clientA2-1 detach:False nothrow:False cmd: ['bash', '-c', "/usr/bin/clickhouse client --host server --query 'SELECT * FROM test_allowed_client_hosts'"] Command:[docker exec roottestallowedclienthosts-gw8-clientA2-1 bash -c /usr/bin/clickhouse client --host server --query 'SELECT * FROM test_allowed_client_hosts'] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT name FROM system.databases WHERE name = 'db1' on node5 http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlQueriesDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_3' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node3 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT * FROM test.tbl2 ORDER BY p on node http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS test_alter_codec_pk on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stderr:Code: 194. DB::Exception: Received from server:9000. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name. Stderr: Stderr:If you use ClickHouse Cloud, the password can be reset at https://clickhouse.cloud/ Stderr:on the settings page for the corresponding service. Stderr: Stderr:If you have installed ClickHouse and forgot password you can reset it in the configuration file. Stderr:The password for default user is typically located at /etc/clickhouse-server/users.d/default-password.xml Stderr:and deleting this file will reset the password. Stderr:See also /etc/clickhouse-server/users.xml on the server where ClickHouse is installed. Stderr: Stderr:. (REQUIRED_PASSWORD) Stderr: Exitcode:194 Command:[docker compose --env-file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/.env --project-name roottestallowedclienthosts-gw8 --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/docker-compose.yml stop --timeout 20] [gw8] PASSED test_allowed_client_hosts/test.py::test_denied_host Executing query select length(thread_ids) from system.query_log where current_database = currentDatabase() and type = 'QueryFinish' and query_id = 'test_concurrent_threads_soft_limit_3' order by query_start_time_microseconds desc limit 1 on node3 Executing query SELECT name FROM system.tables WHERE database = 'db1' AND name = 'test_table' on node5 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None Stdout:8 http://localhost:None "GET /v1.46/containers/c42f6ec293ba833e580d0534e6ac1fe577067a40eae6f6120ed54cb30999fa0a/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.3.7... http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/409a21b272c35678f4611a3421e780eaefcc3be86e4aca976e23985258fd0cb5/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/.env --project-name roottestaltercodec-gw9 --file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/docker-compose.yml stop --timeout 20] [gw9] PASSED test_alter_codec/test.py::test_alter_codec_pk http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/409a21b272c35678f4611a3421e780eaefcc3be86e4aca976e23985258fd0cb5/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT name FROM system.columns WHERE database = 'db1' AND table = 'test_table' on node5 Executing query ALTER TABLE test.tbl2 UPDATE k=10 WHERE 1 on node http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None [gw0] PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_1 test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_50 Executing query SELECT count(*) FROM numbers_mt(10000000) on node2 http://localhost:None "GET /v1.46/containers/409a21b272c35678f4611a3421e780eaefcc3be86e4aca976e23985258fd0cb5/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.3.5... http://localhost:None "GET /v1.46/containers/roottestbackuprestoreoncluster-gw2-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/efb20d69f0d73b5e4cb1f28b90f38c554dd5a4d408789785e18f9f86d6e02056/json HTTP/1.1" 200 None ClickHouse node3 started Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/dd846a99b9d4935bc46662b276a9673f8f46464e0add43ddd9e30fbefaa48e27/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.10.5... http://localhost:None "GET /v1.46/containers/roottestchecktable-gw3-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b4b9b0acb3bef4143713833fa947505596bbebdfaf6202ba3a4d6b59bdebb50d/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts WHERE database = 'db1' AND table = 'test_table' on node5 Executing query SELECT sleep(2) on node http://localhost:None "GET /v1.46/containers/b4b9b0acb3bef4143713833fa947505596bbebdfaf6202ba3a4d6b59bdebb50d/json HTTP/1.1" 200 None ClickHouse node2 started Executing query CREATE DATABASE db1 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts_columns WHERE database = 'db1' AND table = 'test_table' on node5 Executing query INSERT INTO tbl VALUES (1) on node1 Executing query SYSTEM FLUSH LOGS on node2 Executing query CREATE TABLE db1.table1 (date Date, id UInt32, value Int32) ENGINE = MergeTree() PARTITION BY toYYYYMM(date) ORDER BY id SETTINGS min_bytes_for_wide_part=0 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query INSERT INTO db1.table1 VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 Executing query SELECT name FROM system.databases WHERE name = 'db1' on node5 Executing query BACKUP TABLE tbl TO Disk('backups', '1') ASYNC on node1 via HTTP interface Starting new HTTP connection (1): 172.16.3.6:8123 http://172.16.3.6:8123 "GET /?query=BACKUP+TABLE+tbl++TO+Disk%28%27backups%27%2C+%271%27%29+ASYNC HTTP/1.1" 200 None Executing query BACKUP TABLE tbl TO Disk('backups', '1') ASYNC on node2 via HTTP interface Starting new HTTP connection (1): 172.16.3.7:8123 http://172.16.3.7:8123 "GET /?query=BACKUP+TABLE+tbl++TO+Disk%28%27backups%27%2C+%271%27%29+ASYNC HTTP/1.1" 200 None Executing query SELECT status FROM system.backups WHERE id IN ['f5b3bfa3-7a38-4d29-b504-0a568d063457', '9a0ccaf5-0a29-44f5-827b-4c2f787533a9'] AND status == 'CREATING_BACKUP' on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SYSTEM STOP MERGES db1.table1 on node1 Stdout:8 Executing query SELECT name FROM system.tables WHERE database = 'db1' AND name = 'test_table' on node5 Executing query SELECT status FROM system.backups WHERE id IN ['f5b3bfa3-7a38-4d29-b504-0a568d063457', '9a0ccaf5-0a29-44f5-827b-4c2f787533a9'] AND status == 'CREATING_BACKUP' on node2 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query CREATE TABLE db1.table2 (date Date, id UInt32, value Int32) ENGINE = MergeTree() PARTITION BY toYYYYMM(date) ORDER BY id SETTINGS min_bytes_for_wide_part=0 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT name FROM system.columns WHERE database = 'db1' AND table = 'test_table' on node5 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT count() FROM system.backups WHERE id IN ['f5b3bfa3-7a38-4d29-b504-0a568d063457', '9a0ccaf5-0a29-44f5-827b-4c2f787533a9'] AND status == 'BACKUP_CREATED' on node1 Executing query INSERT INTO db1.table2 VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts WHERE database = 'db1' AND table = 'test_table' on node5 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlSlotsGranted'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 Executing query SELECT count() FROM system.backups WHERE id IN ['f5b3bfa3-7a38-4d29-b504-0a568d063457', '9a0ccaf5-0a29-44f5-827b-4c2f787533a9'] AND status == 'BACKUP_CREATED' on node2 Executing query SYSTEM STOP MERGES db1.table2 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts_columns WHERE database = 'db1' AND table = 'test_table' on node5 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlSlotsDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 Executing query SELECT count() FROM system.backups WHERE id IN ['f5b3bfa3-7a38-4d29-b504-0a568d063457', '9a0ccaf5-0a29-44f5-827b-4c2f787533a9'] AND status == 'BACKUP_FAILED' on node1 Executing query CREATE DATABASE db2 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None [gw4] PASSED test_config_substitutions/test.py::test_allow_databases test_config_substitutions/test.py::test_config Executing query select value from system.settings where name = 'max_query_size' on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlSlotsAcquired'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 Executing query SELECT count() FROM system.backups WHERE id IN ['f5b3bfa3-7a38-4d29-b504-0a568d063457', '9a0ccaf5-0a29-44f5-827b-4c2f787533a9'] AND status == 'BACKUP_FAILED' on node2 Executing query CREATE TABLE db2.table1 (date Date, id UInt32, value Int32) ENGINE = MergeTree() PARTITION BY toYYYYMM(date) ORDER BY id SETTINGS min_bytes_for_wide_part=0 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', '\n cat > /var/lib/clickhouse/access/4d72577c-914e-43b7-a202-69ff5ef7dfd3.sql << EOF\nATTACH USER \\`4d72577c-914e-43b7-a202-69ff5ef7dfd3\\`;\nATTACH GRANT CREATE FUNCTION, SELECT ON mydb.* TO \\`4d72577c-914e-43b7-a202-69ff5ef7dfd3\\`;\nEOF'] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c cat > /var/lib/clickhouse/access/4d72577c-914e-43b7-a202-69ff5ef7dfd3.sql << EOF ATTACH USER \`4d72577c-914e-43b7-a202-69ff5ef7dfd3\`; ATTACH GRANT CREATE FUNCTION, SELECT ON mydb.* TO \`4d72577c-914e-43b7-a202-69ff5ef7dfd3\`; EOF] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/access/need_rebuild_lists.mark'] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c touch /var/lib/clickhouse/access/need_rebuild_lists.mark] run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestaccessforfunctions-gw7-instance-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/ea1fb336123fc64e0c209af4832be48ddaf35a87c6e7b6c319d4c133259b10ff/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/ea1fb336123fc64e0c209af4832be48ddaf35a87c6e7b6c319d4c133259b10ff/json HTTP/1.1" 200 586 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query select ProfileEvents['ConcurrencyControlSlotsAcquiredNonCompeting'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 Executing query INSERT INTO db2.table1 VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 Executing query select value from system.settings where name = 'max_query_size' on node2 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT * FROM test.tbl2 ORDER BY p on node http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlQueriesDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 Executing query SYSTEM STOP MERGES db2.table1 on node1 Executing query select value from system.settings where name = 'max_query_size' on node3 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query RESTORE TABLE tbl FROM Disk('backups', '1') on node1 Executing query DROP TABLE IF EXISTS test.tbl2 on node http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stderr: Container roottestaltercodec-gw9-node1-1 Stopping Stderr: Container roottestaltercodec-gw9-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/.env --project-name roottestaltercodec-gw9 --file /ClickHouse/tests/integration/test_alter_codec/_instances-0-gw9/node1/docker-compose.yml down --volumes] Executing query CREATE TABLE db2.table2 (date Date, id UInt32, value Int32) ENGINE = MergeTree() PARTITION BY toYYYYMM(date) ORDER BY id SETTINGS min_bytes_for_wide_part=0 on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None [gw6] PASSED test_backup_restore/test.py::test_attach_partition test_backup_restore/test.py::test_replace_partition Executing query CREATE TABLE test.tbl3 AS test.tbl on node Executing query select value from system.settings where name = 'max_query_size' on node4 Executing query select length(thread_ids) from system.query_log where current_database = currentDatabase() and type = 'QueryFinish' and query_id = 'test_concurrent_threads_soft_limit_2' order by query_start_time_microseconds desc limit 1 on node2 Executing query SELECT * FROM tbl on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query INSERT INTO db2.table2 VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 Executing query INSERT INTO test.tbl3(p, k) VALUES(toDate(3), 3) on node Executing query select value from system.settings where name = 'max_query_size' on node6 Stdout:778 Clickhouse process running. run container_id:roottestaccessforfunctions-gw7-instance-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestaccessforfunctions-gw7-instance-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] [gw0] PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_50 test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_limit_reached Executing query SELECT count(*) FROM numbers_mt(1e11) settings max_threads=100 on node4 Executing query SELECT sum(length(thread_ids)) FROM system.processes on node4 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None [gw2] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 Stdout:778 Executing query select 20 on instance http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SYSTEM STOP MERGES db2.table2 on node1 Executing query INSERT INTO test.tbl3(p, k) VALUES(toDate(4), 4) on node Executing query select value from system.settings where name = 'max_query_size' on node7 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stderr: Container roottestaltercodec-gw9-node1-1 Stopping Stderr: Container roottestaltercodec-gw9-node1-1 Stopped Stderr: Container roottestaltercodec-gw9-node1-1 Removing Stderr: Container roottestaltercodec-gw9-node1-1 Removed Stderr: Network roottestaltercodec-gw9_default Removing Stderr: Network roottestaltercodec-gw9_default Removed Cleanup called Docker networks for project roottestaltercodec-gw9 are NETWORK ID NAME DRIVER SCOPE Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Docker containers for project roottestaltercodec-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestaltercodec-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestaltercodec-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Unstopped containers: {} No running containers for project: roottestaltercodec-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query SELECT sum(length(thread_ids)) FROM system.processes on node4 Stdout:1 Volumes pruned: 1 test_attach_table_normalizer/test.py::test_attach_substr Running tests in /ClickHouse/tests/integration/test_attach_table_normalizer/test.py Cluster start called. is_up=False Docker networks for project roottestattachtablenormalizer-gw9 are NETWORK ID NAME DRIVER SCOPE Executing query CREATE DATABASE db3 on node1 Executing query INSERT INTO test.tbl3(p, k) VALUES(toDate(33), 33) on node Executing query select value from system.settings where name = 'max_threads' on node7 Docker containers for project roottestattachtablenormalizer-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Docker volumes for project roottestattachtablenormalizer-gw9 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestattachtablenormalizer-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestattachtablenormalizer-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestattachtablenormalizer-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestattachtablenormalizer-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestattachtablenormalizer-gw9 Trying to prune unused networks... http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query CREATE TABLE db3.table1 (date Date, id UInt32, value Int32) ENGINE = MergeTree() PARTITION BY toYYYYMM(date) ORDER BY id SETTINGS min_bytes_for_wide_part=0 on node1 Stdout:1 Volumes pruned: 1 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/database Setup logs dir /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/.env --project-name roottestattachtablenormalizer-gw9 --file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/docker-compose.yml pull] Executing query INSERT INTO test.tbl3(p, k) VALUES(toDate(34), 34) on node Executing query select value from system.settings where name = 'max_query_size' on node8 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT count(*) FROM numbers_mt(10000000) settings max_threads=6 on node4 Executing query select 20 on instance http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Stopping Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Stopping Stderr: Container roottestallowedclienthosts-gw8-server-1 Stopping Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Stopping Stderr: Container roottestallowedclienthosts-gw8-server-1 Stopped Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Stopped Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Stopped Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/.env --project-name roottestallowedclienthosts-gw8 --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/server/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA1/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA2/docker-compose.yml --file /ClickHouse/tests/integration/test_allowed_client_hosts/_instances-0-gw8/clientA3/docker-compose.yml down --volumes] Executing query INSERT INTO db3.table1 VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SELECT * FROM test.tbl3 ORDER BY p on node [gw4] PASSED test_config_substitutions/test.py::test_config test_config_substitutions/test.py::test_config_from_env_overrides run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'test ! -f /etc/clickhouse-server/users.d/000-users_with_env_subst.xml || mv --no-clobber /etc/clickhouse-server/users.d/000-users_with_env_subst.xml /tmp/000-users_with_env_subst_78f31cb4-616c-4416-8a38-5ff45c5f4cca.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c test ! -f /etc/clickhouse-server/users.d/000-users_with_env_subst.xml || mv --no-clobber /etc/clickhouse-server/users.d/000-users_with_env_subst.xml /tmp/000-users_with_env_subst_78f31cb4-616c-4416-8a38-5ff45c5f4cca.xml] Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo \'\n\n \n \n \n 100\n \n \n \n \n \n default\n default\n \n\n \n \n \n\n\' > /etc/clickhouse-server/users.d/000-users_with_env_subst.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c echo ' 100 default default ' > /etc/clickhouse-server/users.d/000-users_with_env_subst.xml] Executing query SHOW GRANTS FOR `4d72577c-914e-43b7-a202-69ff5ef7dfd3` on instance http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SYSTEM RELOAD CONFIG on node7 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query SYSTEM STOP MERGES db3.table1 on node1 copy from /var/lib/clickhouse/shadow/*/data/test/tbl to /var/lib/clickhouse/data/test/tbl3/detached run container_id:roottestbackuprestore-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cp -r /var/lib/clickhouse/shadow/*/data/test/tbl -T /var/lib/clickhouse/data/test/tbl3/detached'] Command:[docker exec roottestbackuprestore-gw6-node-1 bash -c cp -r /var/lib/clickhouse/shadow/*/data/test/tbl -T /var/lib/clickhouse/data/test/tbl3/detached] Executing query ALTER TABLE test.tbl3 REPLACE PARTITION 197002 FROM test.tbl on node http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/.env --project-name roottestaccessforfunctions-gw7 --file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/docker-compose.yml stop --timeout 20] [gw7] PASSED test_access_for_functions/test.py::test_ignore_obsolete_grant_on_database Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None Executing query CREATE TABLE db3.table2 (date Date, id UInt32, value Int32) ENGINE = MergeTree() PARTITION BY toYYYYMM(date) ORDER BY id SETTINGS min_bytes_for_wide_part=0 on node1 Executing query SELECT sleep(2) on node run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'test ! -f /tmp/000-users_with_env_subst_78f31cb4-616c-4416-8a38-5ff45c5f4cca.xml || mv /tmp/000-users_with_env_subst_78f31cb4-616c-4416-8a38-5ff45c5f4cca.xml /etc/clickhouse-server/users.d/000-users_with_env_subst.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c test ! -f /tmp/000-users_with_env_subst_78f31cb4-616c-4416-8a38-5ff45c5f4cca.xml || mv /tmp/000-users_with_env_subst_78f31cb4-616c-4416-8a38-5ff45c5f4cca.xml /etc/clickhouse-server/users.d/000-users_with_env_subst.xml] http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'test ! -f /etc/clickhouse-server/users.d/000-users_with_env_subst.xml || mv --no-clobber /etc/clickhouse-server/users.d/000-users_with_env_subst.xml /tmp/000-users_with_env_subst_31cb941b-4656-488b-a974-bb9f5040cfe0.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c test ! -f /etc/clickhouse-server/users.d/000-users_with_env_subst.xml || mv --no-clobber /etc/clickhouse-server/users.d/000-users_with_env_subst.xml /tmp/000-users_with_env_subst_31cb941b-4656-488b-a974-bb9f5040cfe0.xml] run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo \'\n\n \n \n \n 1\n \n \n \n \n \n default\n default\n \n\n \n \n \n\n\' > /etc/clickhouse-server/users.d/000-users_with_env_subst.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c echo ' 1 default default ' > /etc/clickhouse-server/users.d/000-users_with_env_subst.xml] Executing query SYSTEM RELOAD CONFIG on node7 http://localhost:None "GET /v1.46/containers/bdfb38d6f8b1fe65a60136b3f9e0ad49acba8af802a40474795a7f051fe22196/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestclusterallreplicas-gw1-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestclusterallreplicas-gw1-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.9.2... http://localhost:None "GET /v1.46/containers/roottestclusterallreplicas-gw1-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/705563933b918b7ec0b61f84d19991f729b8976f23b060a976c911d3a1504338/json HTTP/1.1" 200 None ClickHouse node2 started Executing query SELECT hostName() FROM clusterAllReplicas('one_shard_two_nodes', system.one) ORDER BY ALL on node1 Executing query INSERT INTO db3.table2 VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 Stderr: Container roottestallowedclienthosts-gw8-server-1 Stopping Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Stopping Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Stopping Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Stopping Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Stopped Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Removing Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Stopped Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Removing Stderr: Container roottestallowedclienthosts-gw8-server-1 Stopped Stderr: Container roottestallowedclienthosts-gw8-server-1 Removing Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Stopped Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Removing Stderr: Container roottestallowedclienthosts-gw8-clientA3-1 Removed Stderr: Container roottestallowedclienthosts-gw8-clientA1-1 Removed Stderr: Container roottestallowedclienthosts-gw8-clientA2-1 Removed Stderr: Container roottestallowedclienthosts-gw8-server-1 Removed Stderr: Network roottestallowedclienthosts-gw8_default Removing Stderr: Network roottestallowedclienthosts-gw8_default Removed Cleanup called Docker networks for project roottestallowedclienthosts-gw8 are NETWORK ID NAME DRIVER SCOPE test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Docker containers for project roottestallowedclienthosts-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestallowedclienthosts-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestallowedclienthosts-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Stderr: Container roottestaccessforfunctions-gw7-instance-1 Stopping Stderr: Container roottestaccessforfunctions-gw7-instance-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/.env --project-name roottestaccessforfunctions-gw7 --file /ClickHouse/tests/integration/test_access_for_functions/_instances-0-gw7/instance/docker-compose.yml down --volumes] Unstopped containers: {} No running containers for project: roottestallowedclienthosts-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Running tests in /ClickHouse/tests/integration/test_attach_partition_with_large_destination/test.py Cluster start called. is_up=False test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Atomic] Docker networks for project roottestattachpartitionwithlargedestination-gw8 are NETWORK ID NAME DRIVER SCOPE Executing query SYSTEM FLUSH LOGS on node4 Docker containers for project roottestattachpartitionwithlargedestination-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestattachpartitionwithlargedestination-gw8 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestattachpartitionwithlargedestination-gw8 are NETWORK ID NAME DRIVER SCOPE Executing query SYSTEM STOP MERGES db3.table2 on node1 run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'test ! -f /tmp/000-users_with_env_subst_31cb941b-4656-488b-a974-bb9f5040cfe0.xml || mv /tmp/000-users_with_env_subst_31cb941b-4656-488b-a974-bb9f5040cfe0.xml /etc/clickhouse-server/users.d/000-users_with_env_subst.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c test ! -f /tmp/000-users_with_env_subst_31cb941b-4656-488b-a974-bb9f5040cfe0.xml || mv /tmp/000-users_with_env_subst_31cb941b-4656-488b-a974-bb9f5040cfe0.xml /etc/clickhouse-server/users.d/000-users_with_env_subst.xml] Docker containers for project roottestattachpartitionwithlargedestination-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query SELECT hostName(), * FROM clusterAllReplicas("one_shard_two_nodes", system.one) ORDER BY ALL on node1 Docker volumes for project roottestattachpartitionwithlargedestination-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestattachpartitionwithlargedestination-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] [gw4] PASSED test_config_substitutions/test.py::test_config_from_env_overrides test_config_substitutions/test.py::test_config_merge_from_env_overrides Executing query SYSTEM RELOAD CONFIG on node7 Unstopped containers: {} No running containers for project: roottestattachpartitionwithlargedestination-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_attach_partition_with_large_destination/configs/config.xml'] to /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/database Setup logs dir /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/.env --project-name roottestattachpartitionwithlargedestination-gw8 --file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml pull] Executing query INSERT INTO tbl VALUES (1) on node1 Executing query SELECT path FROM system.parts WHERE database = 'db1' AND table = 'table2' AND name = '201902_1_1_0' on node1 Executing query SELECT hostName() FROM cluster('one_shard_two_nodes', system.one) on node1 Executing query SELECT value FROM system.server_settings WHERE name='max_thread_pool_size' on node7 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '2') ASYNC on node1 via HTTP interface Starting new HTTP connection (1): 172.16.3.6:8123 http://172.16.3.6:8123 "GET /?query=BACKUP+TABLE+tbl+ON+CLUSTER+%27cluster%27+TO+Disk%28%27backups%27%2C+%272%27%29+ASYNC HTTP/1.1" 200 None Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '2') ASYNC on node2 via HTTP interface Starting new HTTP connection (1): 172.16.3.7:8123 http://172.16.3.7:8123 "GET /?query=BACKUP+TABLE+tbl+ON+CLUSTER+%27cluster%27+TO+Disk%28%27backups%27%2C+%272%27%29+ASYNC HTTP/1.1" 200 None Executing query SELECT status FROM system.backups WHERE id IN ['3ab4bf75-5520-4ea4-8d85-99732aad2114', '0b787d14-2437-467b-b1fb-40a9ad35944b'] AND status == 'CREATING_BACKUP' on node1 run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm -r /var/lib/clickhouse/store/a20/a20226f8-e745-4ed2-bba4-a6589c7f3892/201902_1_1_0//checksums.txt'] Command:[docker exec --privileged roottestchecktable-gw3-node1-1 bash -c rm -r /var/lib/clickhouse/store/a20/a20226f8-e745-4ed2-bba4-a6589c7f3892/201902_1_1_0//checksums.txt] Stderr: Container roottestaccessforfunctions-gw7-instance-1 Stopping Stderr: Container roottestaccessforfunctions-gw7-instance-1 Stopped Stderr: Container roottestaccessforfunctions-gw7-instance-1 Removing Stderr: Container roottestaccessforfunctions-gw7-instance-1 Removed Stderr: Network roottestaccessforfunctions-gw7_default Removing Stderr: Network roottestaccessforfunctions-gw7_default Removed Cleanup called Executing query CREATE TABLE db1.table_memory (date Date, id UInt32, value Int32) ENGINE = Memory() on node1 Docker networks for project roottestaccessforfunctions-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestaccessforfunctions-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query SELECT hostName() FROM cluster('one_shard_two_nodes', system.one) on node2 Docker volumes for project roottestaccessforfunctions-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestaccessforfunctions-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'test ! -f /etc/clickhouse-server/config.d/010-server_with_env_subst.xml || mv --no-clobber /etc/clickhouse-server/config.d/010-server_with_env_subst.xml /tmp/010-server_with_env_subst_07af78b7-67bc-4cb8-ac76-4e88a401029e.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c test ! -f /etc/clickhouse-server/config.d/010-server_with_env_subst.xml || mv --no-clobber /etc/clickhouse-server/config.d/010-server_with_env_subst.xml /tmp/010-server_with_env_subst_07af78b7-67bc-4cb8-ac76-4e88a401029e.xml] Unstopped containers: {} No running containers for project: roottestaccessforfunctions-gw7 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo \'\n\n 9000\n\n\' > /etc/clickhouse-server/config.d/010-server_with_env_subst.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c echo ' 9000 ' > /etc/clickhouse-server/config.d/010-server_with_env_subst.xml] Stdout:1 Volumes pruned: 1 test_alter_on_mixed_type_cluster/test.py::test_alter_on_cluter_non_replicated Running tests in /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/test.py Cluster start called. is_up=False Docker networks for project roottestalteronmixedtypecluster-gw7 are NETWORK ID NAME DRIVER SCOPE Executing query SYSTEM RELOAD CONFIG on node7 Docker containers for project roottestalteronmixedtypecluster-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestalteronmixedtypecluster-gw7 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestalteronmixedtypecluster-gw7 are NETWORK ID NAME DRIVER SCOPE Executing query INSERT INTO db1.table_memory VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 Docker containers for project roottestalteronmixedtypecluster-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestalteronmixedtypecluster-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestalteronmixedtypecluster-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] [gw1] PASSED test_cluster_all_replicas/test.py::test_cluster test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[one_shard_three_nodes] Executing query SELECT hostName() FROM clusterAllReplicas('one_shard_three_nodes', system.one) ORDER BY ALL settings skip_unavailable_shards=0 on node1 Unstopped containers: {} No running containers for project: roottestalteronmixedtypecluster-gw7 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/database Setup logs dir /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/database Setup logs dir /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/database Setup logs dir /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node4 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/configs/config.d Setup database dir /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/database Setup logs dir /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/.env --project-name roottestalteronmixedtypecluster-gw7 --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/docker-compose.yml pull] Executing query SYSTEM STOP MERGES db1.table_memory on node1 run container_id:roottestconfigsubstitutions-gw4-node7-1 detach:False nothrow:False cmd: ['bash', '-c', 'test ! -f /tmp/010-server_with_env_subst_07af78b7-67bc-4cb8-ac76-4e88a401029e.xml || mv /tmp/010-server_with_env_subst_07af78b7-67bc-4cb8-ac76-4e88a401029e.xml /etc/clickhouse-server/config.d/010-server_with_env_subst.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node7-1 bash -c test ! -f /tmp/010-server_with_env_subst_07af78b7-67bc-4cb8-ac76-4e88a401029e.xml || mv /tmp/010-server_with_env_subst_07af78b7-67bc-4cb8-ac76-4e88a401029e.xml /etc/clickhouse-server/config.d/010-server_with_env_subst.xml] run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestconfigsubstitutions-gw4-node3-1 bash -c ps -C clickhouse] [gw4] PASSED test_config_substitutions/test.py::test_config_merge_from_env_overrides test_config_substitutions/test.py::test_config_multiple_zk_substitutions Stdout: PID TTY TIME CMD Stdout: 8 ? 00:00:01 clickhouse run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestconfigsubstitutions-gw4-node3-1 bash -c pkill clickhouse] Executing query CREATE TABLE db1.table_log (date Date, id UInt32, value Int32) ENGINE = TinyLog() on node1 run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT status FROM system.backups WHERE id IN ['3ab4bf75-5520-4ea4-8d85-99732aad2114', '0b787d14-2437-467b-b1fb-40a9ad35944b'] AND status == 'CREATING_BACKUP' on node1 Stdout:8 Executing query INSERT INTO db1.table_log VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 Executing query SELECT status FROM system.backups WHERE id IN ['3ab4bf75-5520-4ea4-8d85-99732aad2114', '0b787d14-2437-467b-b1fb-40a9ad35944b'] AND status == 'CREATING_BACKUP' on node2 Executing query SYSTEM STOP MERGES db1.table_log on node1 Executing query SELECT count() FROM system.backups WHERE id IN ['3ab4bf75-5520-4ea4-8d85-99732aad2114', '0b787d14-2437-467b-b1fb-40a9ad35944b'] AND status == 'BACKUP_CREATED' on node1 Executing query SYSTEM ENABLE FAILPOINT check_table_query_delay_for_part on node1 Executing query CHECK ALL TABLES on node1 Executing query DROP TABLE db3.table2 on node1 Executing query DROP DATABASE db2 on node1 Executing query SELECT count() FROM system.backups WHERE id IN ['3ab4bf75-5520-4ea4-8d85-99732aad2114', '0b787d14-2437-467b-b1fb-40a9ad35944b'] AND status == 'BACKUP_CREATED' on node2 Executing query select ProfileEvents['ConcurrencyControlSlotsGranted'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_4' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node4 Executing query SELECT * FROM test.tbl3 ORDER BY p on node Executing query SELECT count() FROM system.backups WHERE id IN ['3ab4bf75-5520-4ea4-8d85-99732aad2114', '0b787d14-2437-467b-b1fb-40a9ad35944b'] AND status == 'BACKUP_FAILED' on node1 Stderr: node Skipped - Image is already being pulled by zoo1 Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: zoo1 Pulling Stderr: zoo1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper1/log', '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper1/config', '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper1/coordination', '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper2/log', '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper2/config', '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper2/coordination', '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper3/log', '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper3/config', '/ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/keeper3/coordination'] Command:[docker compose --project-name roottestattachpartitionwithlargedestination-gw8 --env-file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Executing query ALTER TABLE test.tbl3 UPDATE k=10 WHERE 1 on node run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Stderr: node1 Skipped - Image is already being pulled by node4 Stderr: zoo1 Skipped - Image is already being pulled by node4 Stderr: zoo2 Skipped - Image is already being pulled by node4 Stderr: zoo3 Skipped - Image is already being pulled by node4 Stderr: node2 Skipped - Image is already being pulled by node4 Stderr: node3 Skipped - Image is already being pulled by node4 Stderr: node4 Pulling Stderr: node4 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper1/log', '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper1/config', '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper1/coordination', '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper2/log', '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper2/config', '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper2/coordination', '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper3/log', '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper3/config', '/ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/keeper3/coordination'] Command:[docker compose --project-name roottestalteronmixedtypecluster-gw7 --env-file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Executing query select ProfileEvents['ConcurrencyControlSlotsDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_4' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node4 Executing query SELECT count() FROM system.backups WHERE id IN ['3ab4bf75-5520-4ea4-8d85-99732aad2114', '0b787d14-2437-467b-b1fb-40a9ad35944b'] AND status == 'BACKUP_FAILED' on node2 Executing query SELECT sleep(2) on node Stderr: instance Pulling Stderr: instance Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/.env --project-name roottestauthentication-gw5 --file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/.env --project-name roottestauthentication-gw5 --file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/docker-compose.yml up -d --no-recreate] Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query select ProfileEvents['ConcurrencyControlSlotsAcquired'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_4' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node4 Stderr:time="2025-04-02T02:52:50Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestattachpartitionwithlargedestination-gw8_default Creating Stderr: Network roottestattachpartitionwithlargedestination-gw8_default Created Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Creating Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Creating Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Creating Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Created Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Created Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Created Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Starting Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Starting Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Starting Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Started Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Started Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Started Stderr:time="2025-04-02T02:52:51Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:52:51Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestattachpartitionwithlargedestination-gw8-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/.env --project-name roottestattachtablenormalizer-gw9 --file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/.env --project-name roottestattachtablenormalizer-gw9 --file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/docker-compose.yml up -d --no-recreate] Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query RESTORE TABLE tbl FROM Disk('backups', '2') on node1 Executing query select ProfileEvents['ConcurrencyControlSlotsAcquiredNonCompeting'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_4' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node4 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Stderr:time="2025-04-02T02:52:50Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestalteronmixedtypecluster-gw7_default Creating Stderr: Network roottestalteronmixedtypecluster-gw7_default Created Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Creating Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Creating Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Creating Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Created Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Created Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Created Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Starting Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Starting Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Starting Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Started Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Started Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Started Stderr:time="2025-04-02T02:52:51Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:52:51Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.5.4, port:2181, use_ssl:False Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr: Network roottestauthentication-gw5_default Creating Stderr: Network roottestauthentication-gw5_default Created Stderr: Container roottestauthentication-gw5-instance-1 Creating Stderr: Container roottestauthentication-gw5-instance-1 Created Stderr: Container roottestauthentication-gw5-instance-1 Starting Stderr: Container roottestauthentication-gw5-instance-1 Started ClickHouse instance created get_instance_ip instance_name=instance http://localhost:None "GET /v1.46/containers/roottestauthentication-gw5-instance-1/json HTTP/1.1" 200 None get_instance_ip instance_name=instance http://localhost:None "GET /v1.46/containers/roottestauthentication-gw5-instance-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in instance, ip: 172.16.7.2... http://localhost:None "GET /v1.46/containers/roottestauthentication-gw5-instance-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None Executing query SELECT * FROM tbl on node1 Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlQueriesDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_concurrent_threads_soft_limit_4' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node4 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr: Network roottestattachtablenormalizer-gw9_default Creating Stderr: Network roottestattachtablenormalizer-gw9_default Created Stderr: Container roottestattachtablenormalizer-gw9-node-1 Creating Stderr: Container roottestattachtablenormalizer-gw9-node-1 Created Stderr: Container roottestattachtablenormalizer-gw9-node-1 Starting Stderr: Container roottestattachtablenormalizer-gw9-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestattachtablenormalizer-gw9-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestattachtablenormalizer-gw9-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.8.2... http://localhost:None "GET /v1.46/containers/roottestattachtablenormalizer-gw9-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None [gw3] PASSED test_check_table/test.py::test_check_all_tables test_check_table/test.py::test_check_normal_table_corruption[] Executing query DROP TABLE IF EXISTS non_replicated_mt on node1 Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None Executing query CREATE TABLE non_replicated_mt(date Date, id UInt32, value Int32) ENGINE = MergeTree() PARTITION BY toYYYYMM(date) ORDER BY id ; on node1 http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None Executing query select length(thread_ids) from system.query_log where current_database = currentDatabase() and type = 'QueryFinish' and query_id = 'test_concurrent_threads_soft_limit_4' order by query_start_time_microseconds desc limit 1 on node4 http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query INSERT INTO non_replicated_mt VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None Executing query KILL QUERY WHERE query_id = 'background_query' SYNC on node4 Executing query CHECK TABLE non_replicated_mt PARTITION 201902 on node1 http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestconfigsubstitutions-gw4-node3-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/6aceaa3a3a2b7946122c37697a63261a1a055abdc5504121d09e4253697b320c/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/6aceaa3a3a2b7946122c37697a63261a1a055abdc5504121d09e4253697b320c/json HTTP/1.1" 200 586 http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None Executing query SELECT * FROM test.tbl3 ORDER BY p on node Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None [gw0] PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_limit_reached test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_default Executing query SELECT count(*) FROM numbers_mt(10000000) SETTINGS use_concurrency_control = 0 on node1 http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS test.tbl3 on node http://localhost:None "GET /v1.46/containers/00eb408c3510c295b333103911f465f280cb552341b3fb9913f0f860a3f77b36/json HTTP/1.1" 200 None ClickHouse instance started Executing query CREATE USER sasha on instance http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None Executing query SYSTEM FLUSH LOGS on node1 [gw6] PASSED test_backup_restore/test.py::test_replace_partition test_backup_restore/test.py::test_restore Executing query CREATE TABLE test.tbl1 AS test.tbl on node Executing query CREATE USER masha IDENTIFIED BY 'qwerty' on instance http://localhost:None "GET /v1.46/containers/64540eced4186709ea02542698846be9fc27d1504f6e0dca8c96a2ab54d41d93/json HTTP/1.1" 200 None ClickHouse node started Executing query DROP TABLE IF EXISTS default.file on node test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 copy from /var/lib/clickhouse/shadow/*/data/test/tbl to /var/lib/clickhouse/data/test/tbl1/detached run container_id:roottestbackuprestore-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cp -r /var/lib/clickhouse/shadow/*/data/test/tbl -T /var/lib/clickhouse/data/test/tbl1/detached'] Command:[docker exec roottestbackuprestore-gw6-node-1 bash -c cp -r /var/lib/clickhouse/shadow/*/data/test/tbl -T /var/lib/clickhouse/data/test/tbl1/detached] Executing query SELECT currentUser() on instance Executing query CREATE TABLE default.file(`s` String, `n` UInt8) ENGINE = MergeTree PARTITION BY substring(s, 1, 2) ORDER BY n on node Executing query ALTER TABLE test.tbl1 ATTACH PARTITION 197001 on node Executing query INSERT INTO tbl VALUES (1) on node1 run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT currentUser() on instance Executing query select ProfileEvents['ConcurrencyControlSlotsGranted'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 Executing query DETACH TABLE file on node Stdout:784 Clickhouse process running. run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query ALTER TABLE test.tbl1 ATTACH PARTITION 197002 on node Stdout:784 Executing query select 20 on node3 Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '3') ASYNC on node1 [gw5] PASSED test_authentication/test.py::test_authentication_fail test_authentication/test.py::test_authentication_pass Executing query SELECT currentUser() on instance Executing query select ProfileEvents['ConcurrencyControlSlotsDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/substring/substr/g' /var/lib/clickhouse/metadata/default/file.sql"] Command:[docker exec -u root roottestattachtablenormalizer-gw9-node-1 bash -c sed -i 's/substring/substr/g' /var/lib/clickhouse/metadata/default/file.sql] Executing query SELECT sleep(2) on node Executing query ATTACH TABLE file on node Executing query SELECT path FROM system.parts WHERE database = 'default' AND table = 'non_replicated_mt' AND name = '201902_1_1_0' on node1 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '3') ASYNC on node2 Executing query SELECT currentUser() on instance Executing query DROP TABLE IF EXISTS default.file on node [gw9] PASSED test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart Executing query select ProfileEvents['ConcurrencyControlSlotsAcquired'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm -r /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201902_1_1_0//checksums.txt'] Command:[docker exec --privileged roottestchecktable-gw3-node1-1 bash -c rm -r /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201902_1_1_0//checksums.txt] Executing query SELECT status FROM system.backups WHERE id IN ['f9672d59-9f20-41d8-bec3-c19ba526f93e', 'd1fe997c-802b-462f-9e87-4d5737747512'] AND status == 'CREATING_BACKUP' on node1 Executing query CHECK TABLE non_replicated_mt on node1 Executing query SELECT currentUser() on instance Executing query CREATE TABLE default.file(`s` String, `n` UInt8) ENGINE = MergeTree PARTITION BY substring(s, 1, 2) ORDER BY n on node Executing query select ProfileEvents['ConcurrencyControlSlotsAcquiredNonCompeting'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 Executing query select 20 on node3 Executing query SELECT status FROM system.backups WHERE id IN ['f9672d59-9f20-41d8-bec3-c19ba526f93e', 'd1fe997c-802b-462f-9e87-4d5737747512'] AND status == 'CREATING_BACKUP' on node2 Executing query SELECT currentUser() on instance run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/substring/substr/g' /var/lib/clickhouse/metadata/default/file.sql"] Command:[docker exec -u root roottestattachtablenormalizer-gw9-node-1 bash -c sed -i 's/substring/substr/g' /var/lib/clickhouse/metadata/default/file.sql] run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestattachtablenormalizer-gw9-node-1 bash -c ps -C clickhouse] Executing query select ProfileEvents['ConcurrencyControlQueriesDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node1 Executing query SYSTEM RELOAD CONFIG on node3 Stdout: PID TTY TIME CMD Stdout: 8 ? 00:00:01 clickhouse run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -9 clickhouse'] Command:[docker exec -u root roottestattachtablenormalizer-gw9-node-1 bash -c pkill -9 clickhouse] run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestattachtablenormalizer-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Command:[docker compose --env-file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/.env --project-name roottestauthentication-gw5 --file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/docker-compose.yml stop --timeout 20] [gw5] PASSED test_authentication/test.py::test_authentication_pass [gw0] PASSED test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_default test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_soft_limit_defined_50 Executing query SELECT count(*) FROM numbers_mt(10000000) SETTINGS use_concurrency_control = 0 on node2 Executing query SELECT value FROM system.merge_tree_settings WHERE name='min_bytes_for_wide_part' on node3 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT COUNT() FROM non_replicated_mt on node1 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestattachpartitionwithlargedestination-gw8-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.2, port:2181, use_ssl:False Executing query SELECT value FROM system.merge_tree_settings WHERE name='min_rows_for_wide_part' on node3 Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestattachpartitionwithlargedestination-gw8-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Executing query SYSTEM FLUSH LOGS on node2 Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT status FROM system.backups WHERE id IN ['f9672d59-9f20-41d8-bec3-c19ba526f93e', 'd1fe997c-802b-462f-9e87-4d5737747512'] AND status == 'CREATING_BACKUP' on node2 Executing query SELECT path FROM system.parts WHERE database = 'default' AND table = 'non_replicated_mt' AND name = '201902_1_1_0' on node1 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/.env --project-name roottestattachpartitionwithlargedestination-gw8 --file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/.env --project-name roottestattachpartitionwithlargedestination-gw8 --file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate] Executing query SELECT value FROM system.merge_tree_settings WHERE name='merge_max_block_size' on node3 Executing query SELECT count() FROM system.backups WHERE id IN ['f9672d59-9f20-41d8-bec3-c19ba526f93e', 'd1fe997c-802b-462f-9e87-4d5737747512'] AND status == 'BACKUP_CREATED' on node1 run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm -r /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201902_1_1_0//checksums.txt'] Command:[docker exec --privileged roottestchecktable-gw3-node1-1 bash -c rm -r /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201902_1_1_0//checksums.txt] Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT value FROM system.server_settings WHERE name='background_pool_size' on node3 Executing query CHECK TABLE non_replicated_mt PARTITION 201902 on node1 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.5.3, port:2181, use_ssl:False Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Running Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Running Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Running Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Creating Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Created Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Starting Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestattachpartitionwithlargedestination-gw8-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestattachpartitionwithlargedestination-gw8-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestattachpartitionwithlargedestination-gw8-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.5.2, port:2181, use_ssl:False Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestattachtablenormalizer-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT count() FROM system.backups WHERE id IN ['f9672d59-9f20-41d8-bec3-c19ba526f93e', 'd1fe997c-802b-462f-9e87-4d5737747512'] AND status == 'BACKUP_CREATED' on node2 run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestattachtablenormalizer-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query select ProfileEvents['ConcurrencyControlSlotsGranted'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/.env --project-name roottestalteronmixedtypecluster-gw7 --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/.env --project-name roottestalteronmixedtypecluster-gw7 --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/docker-compose.yml up -d --no-recreate] get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestconfigsubstitutions-gw4-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.4.2, port:2181, use_ssl:False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) No clickhouse process running. Start new one. Zookeeper connection established, state: CONNECTED Sending request(xid=1): Create(path='/background_pool_size', data=b'72', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) http://localhost:None "POST /v1.46/containers/roottestattachtablenormalizer-gw9-node-1/exec HTTP/1.1" 201 74 Received response(xid=1): '/background_pool_size' run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', 'test ! -f /etc/clickhouse-server/config.d/config_zk_include_test.xml || mv --no-clobber /etc/clickhouse-server/config.d/config_zk_include_test.xml /tmp/config_zk_include_test_69278f3b-08a7-43d9-8e84-951379675d55.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c test ! -f /etc/clickhouse-server/config.d/config_zk_include_test.xml || mv --no-clobber /etc/clickhouse-server/config.d/config_zk_include_test.xml /tmp/config_zk_include_test_69278f3b-08a7-43d9-8e84-951379675d55.xml] http://localhost:None "POST /v1.46/exec/1050c993849e7b3ef515ae81ba0f37304b23608cf110b9f726fba47904e619ce/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/1050c993849e7b3ef515ae81ba0f37304b23608cf110b9f726fba47904e619ce/json HTTP/1.1" 200 586 run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo \'\n \n \n 44\n \n \n 1\n 1111\n \n\n \n \n \' > /etc/clickhouse-server/config.d/config_zk_include_test.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c echo ' 44 1 1111 ' > /etc/clickhouse-server/config.d/config_zk_include_test.xml] Executing query SYSTEM RELOAD CONFIG on node3 http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None [gw1] PASSED test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[one_shard_three_nodes] test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[two_shards_three_nodes] Executing query SELECT hostName() FROM clusterAllReplicas('two_shards_three_nodes', system.one) ORDER BY ALL settings skip_unavailable_shards=0 on node1 Executing query SELECT count() FROM system.backups WHERE id IN ['f9672d59-9f20-41d8-bec3-c19ba526f93e', 'd1fe997c-802b-462f-9e87-4d5737747512'] AND status == 'BACKUP_FAILED' on node1 http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlSlotsDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Executing query SELECT value FROM system.server_settings WHERE name='background_pool_size' on node3 Executing query SELECT * FROM test.tbl1 ORDER BY p on node http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Executing query SELECT count() FROM system.backups WHERE id IN ['f9672d59-9f20-41d8-bec3-c19ba526f93e', 'd1fe997c-802b-462f-9e87-4d5737747512'] AND status == 'BACKUP_FAILED' on node2 http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlSlotsAcquired'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Running Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Running Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Running Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Creating Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Creating Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Creating Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Creating Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Created Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Created Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Created Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Created Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Starting Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Starting Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Starting Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Starting Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Started Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Started Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Started Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.5.7... http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 run container_id:roottestconfigsubstitutions-gw4-node3-1 detach:False nothrow:False cmd: ['bash', '-c', 'test ! -f /tmp/config_zk_include_test_69278f3b-08a7-43d9-8e84-951379675d55.xml || mv /tmp/config_zk_include_test_69278f3b-08a7-43d9-8e84-951379675d55.xml /etc/clickhouse-server/config.d/config_zk_include_test.xml'] Command:[docker exec roottestconfigsubstitutions-gw4-node3-1 bash -c test ! -f /tmp/config_zk_include_test_69278f3b-08a7-43d9-8e84-951379675d55.xml || mv /tmp/config_zk_include_test_69278f3b-08a7-43d9-8e84-951379675d55.xml /etc/clickhouse-server/config.d/config_zk_include_test.xml] Executing query ALTER TABLE test.tbl1 UPDATE k=10 WHERE 1 on node Executing query select ProfileEvents['ConcurrencyControlSlotsAcquiredNonCompeting'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Sending request(xid=2): Delete(path='/background_pool_size', version=-1) Received response(xid=2): True [gw4] PASSED test_config_substitutions/test.py::test_config_multiple_zk_substitutions test_config_substitutions/test.py::test_include_config Executing query select 1 on node4 http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Executing query SELECT COUNT() FROM non_replicated_mt on node1 Executing query SELECT sleep(2) on node http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestattachtablenormalizer-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:738 Clickhouse process running. run container_id:roottestattachtablenormalizer-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestattachtablenormalizer-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Executing query select ProfileEvents['ConcurrencyControlQueriesDelayed'] from system.query_log where current_database = currentDatabase() and query_id = 'test_use_concurrency_control_2' and type = 'QueryFinish' order by query_start_time_microseconds desc limit 1 on node2 Executing query RESTORE TABLE tbl FROM Disk('backups', '3') on node1 Executing query select 1 on node4 http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Stdout:738 Executing query select 20 on node Executing query SELECT path FROM system.parts WHERE table = 'non_replicated_mt' and name = '201902_1_1_0' AND database = 'default' on node1 http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/.env --project-name roottestconcurrentthreadssoftlimit-gw0 --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/docker-compose.yml stop --timeout 20] [gw0] PASSED test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_soft_limit_defined_50 http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None Executing query select 1 on node4 run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'cd /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201902_1_1_0/ && ls *.bin | head -n 1'] Command:[docker exec roottestchecktable-gw3-node1-1 bash -c cd /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201902_1_1_0/ && ls *.bin | head -n 1] http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/afdb329cb1411d0149b1fe57198152e6a8becbcebd193d8eb434b7add3786539/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE DATABASE db ENGINE=Atomic on node Executing query SELECT * FROM tbl on node1 Stdout:data.bin run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "cd /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201902_1_1_0/ && ls *.bin | head -n 1 | xargs -I{} sh -c 'truncate -s -1 $1' -- {}"] Command:[docker exec --privileged roottestchecktable-gw3-node1-1 bash -c cd /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201902_1_1_0/ && ls *.bin | head -n 1 | xargs -I{} sh -c 'truncate -s -1 $1' -- {}] http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Executing query CHECK TABLE non_replicated_mt on node1 Executing query select 1 on node3 http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Executing query CREATE TABLE db.destination (n UInt64) ENGINE=ReplicatedMergeTree('/test/destination', 'r1') ORDER BY n PARTITION BY n % 2 on node Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None [gw2] PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] Executing query select 1 on node3 http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Executing query select 20 on node http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Executing query CREATE TABLE db.source_1 (n UInt64) ENGINE=ReplicatedMergeTree('/test/source_1', 'r1') ORDER BY n PARTITION BY n % 2 on node http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Executing query select 1 on node3 Command:[docker compose --env-file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/.env --project-name roottestattachtablenormalizer-gw9 --file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/docker-compose.yml stop --timeout 20] [gw9] PASSED test_attach_table_normalizer/test.py::test_attach_substr_restart http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None Executing query INSERT INTO db.source_1 VALUES (1), (2), (3), (4) on node Executing query select 1 on node8 http://localhost:None "GET /v1.46/containers/b80c2b97d7a06d4358dc7a7ee5dc4be539ed3a535fa62246055f6f741ca8efa0/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.5.6... http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9911dd559e820169a0987fb7da5ea6f6c10b1e58635daf066c74a1858b333ff0/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9911dd559e820169a0987fb7da5ea6f6c10b1e58635daf066c74a1858b333ff0/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.5.8... http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e81da164acc5f7232dc2b39cb768788f3a514bcafd676c11b6e49692acf06e99/json HTTP/1.1" 200 None ClickHouse node3 started get_instance_ip instance_name=node4 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node4-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node4 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node4-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node4, ip: 172.16.5.5... Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/roottestalteronmixedtypecluster-gw7-node4-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/de8e5ebc274f7a60764b4fbacfcc4e8da745571ae6cf8d716cecb9b26e5848d0/json HTTP/1.1" 200 None ClickHouse node4 started Executing query CREATE TABLE IF NOT EXISTS test_table_replicated(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/0/sometable', 'node1') ORDER BY id; on node1 Stderr: Container roottestattachtablenormalizer-gw9-node-1 Stopping Stderr: Container roottestattachtablenormalizer-gw9-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/.env --project-name roottestattachtablenormalizer-gw9 --file /ClickHouse/tests/integration/test_attach_table_normalizer/_instances-0-gw9/node/docker-compose.yml down --volumes] Executing query select 1 on node8 Executing query CREATE TABLE db.source_2 (n UInt64) ENGINE=ReplicatedMergeTree('/test/source_2', 'r1') ORDER BY n PARTITION BY n % 2 on node Executing query CREATE TABLE IF NOT EXISTS test_table(date Date, id UInt32, value Int32) ENGINE=MergeTree ORDER BY id on node1 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query select 1 on node8 Executing query CREATE TABLE IF NOT EXISTS test_table_replicated(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/0/sometable', 'node2') ORDER BY id; on node2 Executing query INSERT INTO db.source_2 VALUES (5), (6), (7), (8) on node Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Stderr: Container roottestauthentication-gw5-instance-1 Stopping Stderr: Container roottestauthentication-gw5-instance-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/.env --project-name roottestauthentication-gw5 --file /ClickHouse/tests/integration/test_authentication/_instances-0-gw5/instance/docker-compose.yml down --volumes] [gw4] PASSED test_config_substitutions/test.py::test_include_config Command:[docker compose --env-file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/.env --project-name roottestconfigsubstitutions-gw4 --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/docker-compose.yml stop --timeout 20] Executing query ALTER TABLE db.destination ATTACH PARTITION 0 FROM db.source_1 on node Executing query CREATE TABLE IF NOT EXISTS test_table(date Date, id UInt32, value Int32) ENGINE=MergeTree ORDER BY id on node2 Executing query SELECT * FROM test.tbl1 ORDER BY p on node test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}') on node1 Stderr: Container roottestattachtablenormalizer-gw9-node-1 Stopping Stderr: Container roottestattachtablenormalizer-gw9-node-1 Stopped Stderr: Container roottestattachtablenormalizer-gw9-node-1 Removing Stderr: Container roottestattachtablenormalizer-gw9-node-1 Removed Stderr: Network roottestattachtablenormalizer-gw9_default Removing Stderr: Network roottestattachtablenormalizer-gw9_default Removed Cleanup called Executing query CHECK TABLE non_replicated_mt on node1 Docker networks for project roottestattachtablenormalizer-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestattachtablenormalizer-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestattachtablenormalizer-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestattachtablenormalizer-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestattachtablenormalizer-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_buffer_profile/test.py::test_buffer_profile Running tests in /ClickHouse/tests/integration/test_buffer_profile/test.py Cluster start called. is_up=False Docker networks for project roottestbufferprofile-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbufferprofile-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbufferprofile-gw9 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbufferprofile-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbufferprofile-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbufferprofile-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbufferprofile-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbufferprofile-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query SELECT n FROM db.destination ORDER BY n on node Stdout:1 Volumes pruned: 1 Setup directory for instance: node_default Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/configs/config.d Setup database dir /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/database Setup logs dir /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node_buffer_profile Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_buffer_profile/configs/buffer_profile.xml'] to /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/configs/config.d Setup database dir /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/database Setup logs dir /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Executing query CREATE TABLE IF NOT EXISTS test_table_replicated(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/someotable', 'node3') ORDER BY id; on node3 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/.env --project-name roottestbufferprofile-gw9 --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/docker-compose.yml --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/docker-compose.yml pull] Executing query DROP TABLE IF EXISTS test.tbl1 on node Executing query CREATE TABLE mydb.test (`name` String, `value` UInt32) ENGINE = ReplicatedMergeTree ORDER BY value on node1 Executing query ALTER TABLE db.destination REPLACE PARTITION 0 FROM db.source_2 on node Executing query CREATE TABLE IF NOT EXISTS test_table(date Date, id UInt32, value Int32) ENGINE=MergeTree ORDER BY id on node3 Stderr: Container roottestauthentication-gw5-instance-1 Stopping Stderr: Container roottestauthentication-gw5-instance-1 Stopped Stderr: Container roottestauthentication-gw5-instance-1 Removing Stderr: Container roottestauthentication-gw5-instance-1 Removed Stderr: Network roottestauthentication-gw5_default Removing Stderr: Network roottestauthentication-gw5_default Removed Cleanup called Docker networks for project roottestauthentication-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestauthentication-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/.env --project-name roottestbackuprestore-gw6 --file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/docker-compose.yml stop --timeout 20] [gw6] PASSED test_backup_restore/test.py::test_restore Docker volumes for project roottestauthentication-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestauthentication-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestauthentication-gw5 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_backward_compatibility/test_functions.py::test_aggregate_states Running tests in /ClickHouse/tests/integration/test_backward_compatibility/test_functions.py Cluster start called. is_up=False Docker networks for project roottestbackwardcompatibilityfunctions-gw5 are NETWORK ID NAME DRIVER SCOPE Executing query SELECT n FROM db.destination ORDER BY n on node Docker containers for project roottestbackwardcompatibilityfunctions-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query CREATE TABLE IF NOT EXISTS test_table_replicated(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/someotable', 'node4') ORDER BY id; on node4 Docker volumes for project roottestbackwardcompatibilityfunctions-gw5 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbackwardcompatibilityfunctions-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityfunctions-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query INSERT INTO mydb.test VALUES ('abc', 1), ('def', 2) on node1 Docker volumes for project roottestbackwardcompatibilityfunctions-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityfunctions-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityfunctions-gw5 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: upstream Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: backward Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/.env --project-name roottestbackwardcompatibilityfunctions-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/docker-compose.yml pull] Executing query INSERT INTO non_replicated_mt VALUES (toDate('2019-01-01'), 1, 10), (toDate('2019-01-01'), 2, 12) on node1 Executing query ALTER TABLE db.destination ATTACH PARTITION 0 FROM db.source_2 on node Executing query INSERT INTO mydb.test VALUES ('ghi', 3) on node1 Executing query CREATE TABLE IF NOT EXISTS test_table(date Date, id UInt32, value Int32) ENGINE=MergeTree ORDER BY id on node4 Executing query CHECK TABLE non_replicated_mt PARTITION 201901 on node1 Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Stopping Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Stopping Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Stopping Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Stopping Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Stopped Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Stopped Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Stopped Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/.env --project-name roottestconcurrentthreadssoftlimit-gw0 --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_concurrent_threads_soft_limit/_instances-0-gw0/node4/docker-compose.yml down --volumes] Executing query SELECT n FROM db.destination ORDER BY n on node Executing query BACKUP DATABASE mydb TO Disk('backups', '4') on node1 Executing query INSERT INTO test_table VALUES(toDate('2019-10-01'), 1, 1) on node1 Executing query INSERT INTO test_table VALUES(toDate('2019-10-01'), 1, 1) on node2 run container_id:roottestattachpartitionwithlargedestination-gw8-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table'] Command:[docker exec -u root roottestattachpartitionwithlargedestination-gw8-node-1 bash -c touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table] Executing query DROP DATABASE mydb SYNC on node1 Executing query DROP TABLE db.source_1 SYNC on node Executing query INSERT INTO test_table VALUES(toDate('2019-10-01'), 1, 1) on node3 Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Stopping Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Stopping Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Stopping Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Stopping Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Stopped Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Removing Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Stopped Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Removing Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Stopped Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Removing Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Stopped Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Removing Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node3-1 Removed Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node1-1 Removed Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node2-1 Removed Stderr: Container roottestconcurrentthreadssoftlimit-gw0-node4-1 Removed Stderr: Network roottestconcurrentthreadssoftlimit-gw0_default Removing Stderr: Network roottestconcurrentthreadssoftlimit-gw0_default Removed Cleanup called Docker networks for project roottestconcurrentthreadssoftlimit-gw0 are NETWORK ID NAME DRIVER SCOPE Executing query RESTORE DATABASE mydb FROM Disk('backups', '4') on node1 Docker containers for project roottestconcurrentthreadssoftlimit-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestconcurrentthreadssoftlimit-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestconcurrentthreadssoftlimit-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] run container_id:roottestattachpartitionwithlargedestination-gw8-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table'] Command:[docker exec -u root roottestattachpartitionwithlargedestination-gw8-node-1 bash -c touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table] Unstopped containers: {} No running containers for project: roottestconcurrentthreadssoftlimit-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query DROP TABLE db.source_2 SYNC on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_async_connect_to_multiple_ips/test.py::test Running tests in /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/test.py Cluster start called. is_up=False Docker networks for project roottestasyncconnecttomultipleips-gw0 are NETWORK ID NAME DRIVER SCOPE Executing query INSERT INTO test_table VALUES(toDate('2019-10-01'), 1, 1) on node4 Docker containers for project roottestasyncconnecttomultipleips-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestasyncconnecttomultipleips-gw0 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestasyncconnecttomultipleips-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestasyncconnecttomultipleips-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestasyncconnecttomultipleips-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestasyncconnecttomultipleips-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestasyncconnecttomultipleips-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/configs/listen_host.xml'] to /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/database Setup logs dir /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/configs/listen_host.xml'] to /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/database Setup logs dir /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/.env --project-name roottestasyncconnecttomultipleips-gw0 --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_net.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/docker-compose.yml pull] run container_id:roottestattachpartitionwithlargedestination-gw8-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table'] Command:[docker exec -u root roottestattachpartitionwithlargedestination-gw8-node-1 bash -c touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table] Executing query SELECT path FROM system.parts WHERE table = 'non_replicated_mt' and name = '201901_2_2_0' AND database = 'default' on node1 Executing query SELECT COUNT() FROM test_table on node1 Executing query DROP TABLE db.destination SYNC on node Executing query RESTORE DATABASE mydb FROM Disk('backups', '4') SETTINGS structure_only=true on node1 run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'cd /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201901_2_2_0/ && ls *.bin | head -n 1'] Command:[docker exec roottestchecktable-gw3-node1-1 bash -c cd /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201901_2_2_0/ && ls *.bin | head -n 1] Stdout:data.bin run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "cd /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201901_2_2_0/ && ls *.bin | head -n 1 | xargs -I{} sh -c 'truncate -s -1 $1' -- {}"] Command:[docker exec --privileged roottestchecktable-gw3-node1-1 bash -c cd /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201901_2_2_0/ && ls *.bin | head -n 1 | xargs -I{} sh -c 'truncate -s -1 $1' -- {}] Executing query SELECT COUNT() FROM test_table on node2 Executing query SELECT path FROM system.parts WHERE database = 'default' AND table = 'non_replicated_mt' AND name = '201901_2_2_0' on node1 run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm -r /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201901_2_2_0//checksums.txt'] Command:[docker exec --privileged roottestchecktable-gw3-node1-1 bash -c rm -r /var/lib/clickhouse/store/650/65099c84-3f1d-48cd-ab8f-d5304c2cc34b/201901_2_2_0//checksums.txt] Executing query SYSTEM SYNC REPLICA mydb.test on node1 Executing query CHECK TABLE non_replicated_mt PARTITION 201901 on node1 [gw1] PASSED test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[two_shards_three_nodes] test_cluster_all_replicas/test.py::test_global_in Executing query DROP TABLE IF EXISTS u; on node1 Executing query DROP DATABASE db on node Executing query SELECT COUNT() FROM test_table on node3 Executing query SELECT * FROM mydb.test ORDER BY name on node1 Executing query CREATE TABLE u(uid Int16) ENGINE=Memory as select 0 on node1 [gw8] PASSED test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Atomic] test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Ordinary] Executing query CREATE DATABASE db ENGINE=Ordinary on node Executing query SELECT COUNT() FROM test_table on node4 Executing query CREATE TABLE db.destination (n UInt64) ENGINE=ReplicatedMergeTree('/test/destination', 'r1') ORDER BY n PARTITION BY n % 2 on node Executing query SELECT hostName(), * FROM clusterAllReplicas("one_shard_two_nodes", system.one) where dummy GLOBAL IN u ORDER BY ALL on node1 Executing query DROP DATABASE mydb SYNC on node1 Executing query ALTER TABLE test_table ON CLUSTER 'test_cluster_mixed' MODIFY COLUMN date DateTime on node1 [gw1] PASSED test_cluster_all_replicas/test.py::test_global_in test_cluster_all_replicas/test.py::test_skip_unavailable_replica[one_shard_three_nodes] Executing query SELECT hostName() FROM clusterAllReplicas('one_shard_three_nodes', system.one) ORDER BY ALL settings skip_unavailable_shards=1 on node1 Executing query CREATE TABLE db.source_1 (n UInt64) ENGINE=ReplicatedMergeTree('/test/source_1', 'r1') ORDER BY n PARTITION BY n % 2 on node Executing query RESTORE DATABASE mydb FROM Disk('backups', '4') SETTINGS allow_non_empty_tables=true on node1 Executing query SELECT date FROM test_table on node1 Executing query SELECT date FROM test_table on node2 Executing query INSERT INTO db.source_1 VALUES (1), (2), (3), (4) on node Executing query SYSTEM SYNC REPLICA mydb.test on node1 Executing query SELECT date FROM test_table on node3 Executing query CREATE TABLE db.source_2 (n UInt64) ENGINE=ReplicatedMergeTree('/test/source_2', 'r1') ORDER BY n PARTITION BY n % 2 on node Executing query SELECT * FROM mydb.test ORDER BY name on node1 Executing query SELECT date FROM test_table on node4 [gw3] PASSED test_check_table/test.py::test_check_normal_table_corruption[] test_check_table/test.py::test_check_replicated_table_corruption[-_0-.bin] Executing query DROP TABLE IF EXISTS replicated_mt_1 SYNC on node1 Executing query INSERT INTO db.source_2 VALUES (5), (6), (7), (8) on node Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica Stderr: Container roottestbackuprestore-gw6-node-1 Stopping Stderr: Container roottestbackuprestore-gw6-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/.env --project-name roottestbackuprestore-gw6 --file /ClickHouse/tests/integration/test_backup_restore/_instances-0-gw6/node/docker-compose.yml down --volumes] Executing query CREATE TABLE replicated_mt_1(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/replicated_mt_1__0', 'node1') PARTITION BY toYYYYMM(date) ORDER BY id on node1 Executing query ALTER TABLE db.destination ATTACH PARTITION 0 FROM db.source_1 on node Executing query ALTER TABLE test_table ON CLUSTER 'test_cluster_mixed' MODIFY COLUMN value String on node3 Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query SELECT n FROM db.destination ORDER BY n on node Executing query DROP TABLE IF EXISTS replicated_mt_1 SYNC on node2 Executing query INSERT INTO test_table VALUES(toDateTime('2019-10-02 00:00:00'), 2, 'Hello') on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query ALTER TABLE db.destination REPLACE PARTITION 0 FROM db.source_2 on node Executing query CREATE TABLE replicated_mt_1(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/replicated_mt_1__0', 'node2') PARTITION BY toYYYYMM(date) ORDER BY id on node2 Executing query INSERT INTO test_table VALUES(toDateTime('2019-10-02 00:00:00'), 2, 'Hello') on node2 Stderr: Container roottestbackuprestore-gw6-node-1 Stopping Stderr: Container roottestbackuprestore-gw6-node-1 Stopped Stderr: Container roottestbackuprestore-gw6-node-1 Removing Stderr: Container roottestbackuprestore-gw6-node-1 Removed Stderr: Network roottestbackuprestore-gw6_default Removing Stderr: Network roottestbackuprestore-gw6_default Removed Cleanup called Docker networks for project roottestbackuprestore-gw6 are NETWORK ID NAME DRIVER SCOPE Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Docker containers for project roottestbackuprestore-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackuprestore-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackuprestore-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackuprestore-gw6 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query SELECT n FROM db.destination ORDER BY n on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible Running tests in /ClickHouse/tests/integration/test_backward_compatibility/test_insert_profile_events.py Cluster start called. is_up=False Executing query INSERT INTO test_table VALUES(toDateTime('2019-10-02 00:00:00'), 2, 'Hello') on node3 Docker networks for project roottestbackwardcompatibilityinsertprofileevents-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityinsertprofileevents-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityinsertprofileevents-gw6 are DRIVER VOLUME NAME Cleanup called Executing query INSERT INTO replicated_mt_1 VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 Docker networks for project roottestbackwardcompatibilityinsertprofileevents-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityinsertprofileevents-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityinsertprofileevents-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityinsertprofileevents-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityinsertprofileevents-gw6 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query ALTER TABLE db.destination ATTACH PARTITION 0 FROM db.source_2 on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: upstream_node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: old_node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/.env --project-name roottestbackwardcompatibilityinsertprofileevents-gw6 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/docker-compose.yml pull] Executing query INSERT INTO test_table VALUES(toDateTime('2019-10-02 00:00:00'), 2, 'Hello') on node4 Executing query INSERT INTO replicated_mt_1 VALUES (toDate('2019-01-02'), 3, 10), (toDate('2019-01-02'), 4, 12) on node1 Executing query SELECT n FROM db.destination ORDER BY n on node Executing query SELECT COUNT() FROM test_table on node1 test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes Executing query CREATE TABLE tbl (`x` UInt8, `y` String) ENGINE = MergeTree ORDER BY x on node1 Executing query SYSTEM SYNC REPLICA replicated_mt_1 on node2 run container_id:roottestattachpartitionwithlargedestination-gw8-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table'] Command:[docker exec -u root roottestattachpartitionwithlargedestination-gw8-node-1 bash -c touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table] Executing query CREATE TABLE tbl (`w` Int64) ENGINE = MergeTree ORDER BY w on node2 Executing query SELECT COUNT() FROM test_table on node2 Executing query DROP TABLE db.source_1 SYNC on node Executing query SELECT count() from replicated_mt_1 on node1 Executing query INSERT INTO tbl VALUES (1, 'Don''t'), (2, 'count'), (3, 'your'), (4, 'chickens') on node1 Executing query SELECT COUNT() FROM test_table on node3 run container_id:roottestattachpartitionwithlargedestination-gw8-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table'] Command:[docker exec -u root roottestattachpartitionwithlargedestination-gw8-node-1 bash -c touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table] Executing query DROP TABLE db.source_2 SYNC on node Executing query SELECT count() from replicated_mt_1 on node2 Executing query INSERT INTO tbl VALUES (-333), (-222), (-111), (0), (111) on node2 Executing query SELECT COUNT() FROM test_table on node4 run container_id:roottestattachpartitionwithlargedestination-gw8-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table'] Command:[docker exec -u root roottestattachpartitionwithlargedestination-gw8-node-1 bash -c touch /var/lib/clickhouse/flags/force_drop_table && chmod a=rw /var/lib/clickhouse/flags/force_drop_table] Executing query DROP TABLE db.destination SYNC on node Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '5') on node1 Executing query SELECT name from system.parts where table = 'replicated_mt_1' and partition_id = '201901' and active = 1 on node1 Executing query TRUNCATE TABLE test_table on node1 Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP DATABASE db on node Executing query SELECT path FROM system.parts WHERE table = 'replicated_mt_1' and name = '201901_0_0_0' AND database = 'default' on node1 Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query TRUNCATE TABLE test_table on node2 Command:[docker compose --env-file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/.env --project-name roottestattachpartitionwithlargedestination-gw8 --file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] [gw8] PASSED test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Ordinary] Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'cd /var/lib/clickhouse/store/bb7/bb783ff3-3262-4e08-aca8-40e84ba04469/201901_0_0_0/ && ls *.bin | head -n 1'] Connection dropped: socket connection error: Connection refused Command:[docker exec roottestchecktable-gw3-node1-1 bash -c cd /var/lib/clickhouse/store/bb7/bb783ff3-3262-4e08-aca8-40e84ba04469/201901_0_0_0/ && ls *.bin | head -n 1] Executing query TRUNCATE TABLE test_table on node3 Stdout:data.bin run container_id:roottestchecktable-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "cd /var/lib/clickhouse/store/bb7/bb783ff3-3262-4e08-aca8-40e84ba04469/201901_0_0_0/ && ls *.bin | head -n 1 | xargs -I{} sh -c 'truncate -s -1 $1' -- {}"] Command:[docker exec --privileged roottestchecktable-gw3-node1-1 bash -c cd /var/lib/clickhouse/store/bb7/bb783ff3-3262-4e08-aca8-40e84ba04469/201901_0_0_0/ && ls *.bin | head -n 1 | xargs -I{} sh -c 'truncate -s -1 $1' -- {}] Executing query CHECK TABLE replicated_mt_1 PARTITION 201901 on node1 Executing query TRUNCATE TABLE test_table on node4 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '5') on node2 Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused [gw7] PASSED test_alter_on_mixed_type_cluster/test.py::test_alter_on_cluter_non_replicated test_alter_on_mixed_type_cluster/test.py::test_alter_replicated_on_cluster Executing query INSERT INTO test_table_replicated VALUES(toDate('2019-10-01'), 1, 1) on node1 Executing query INSERT INTO test_table_replicated VALUES(toDate('2019-10-01'), 1, 1) on node3 Executing query SELECT * FROM tbl on node1 Executing query SYSTEM SYNC REPLICA test_table_replicated on node2 Executing query SELECT * FROM tbl on node2 Connection dropped: outstanding heartbeat ping not received Transition to CONNECTING Zookeeper connection lost Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Executing query SYSTEM SYNC REPLICA test_table_replicated on node4 Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes Executing query SYSTEM SYNC REPLICA replicated_mt_1 on node1 Executing query ALTER TABLE test_table_replicated ON CLUSTER 'test_cluster_mixed' MODIFY COLUMN date DateTime on node1 Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query SELECT date FROM test_table_replicated on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query SELECT date FROM test_table_replicated on node2 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query SELECT date FROM test_table_replicated on node3 Executing query SELECT date FROM test_table_replicated on node4 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query ALTER TABLE test_table_replicated ON CLUSTER 'test_cluster_mixed' MODIFY COLUMN value String on node3 [gw1] PASSED test_cluster_all_replicas/test.py::test_skip_unavailable_replica[one_shard_three_nodes] test_cluster_all_replicas/test.py::test_skip_unavailable_replica[two_shards_three_nodes] Executing query SELECT hostName() FROM clusterAllReplicas('two_shards_three_nodes', system.one) ORDER BY ALL settings skip_unavailable_shards=1 on node1 test_backup_restore_on_cluster/test.py::test_empty_replicated_table Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8, y String) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Executing query INSERT INTO test_table_replicated VALUES(toDateTime('2019-10-02 00:00:00'), 2, 'Hello') on node2 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '6') SETTINGS replica_num=1 on node1 Stderr: zoo1 Skipped - Image is already being pulled by node1 Stderr: zoo2 Skipped - Image is already being pulled by node1 Stderr: zoo3 Skipped - Image is already being pulled by node1 Stderr: node2 Skipped - Image is already being pulled by node1 Stderr: node1 Pulling Stderr: node1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper1/log', '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper1/config', '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper1/coordination', '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper2/log', '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper2/config', '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper2/coordination', '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper3/log', '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper3/config', '/ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/keeper3/coordination'] Command:[docker compose --project-name roottestasyncconnecttomultipleips-gw0 --env-file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_net.yml --verbose up -d] Stderr: node_buffer_profile Skipped - Image is already being pulled by node_default Stderr: node_default Pulling Stderr: node_default Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/.env --project-name roottestbufferprofile-gw9 --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/docker-compose.yml --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/.env --project-name roottestbufferprofile-gw9 --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/docker-compose.yml --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/docker-compose.yml up -d --no-recreate] Executing query INSERT INTO test_table_replicated VALUES(toDateTime('2019-10-02 00:00:00'), 2, 'Hello') on node4 Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query SYSTEM SYNC REPLICA test_table_replicated on node1 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '6') on node1 Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Stopping Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Stopped Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Stopping Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Stopping Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Stopping Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Stopped Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Stopped Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/.env --project-name roottestattachpartitionwithlargedestination-gw8 --file /ClickHouse/tests/integration/test_attach_partition_with_large_destination/_instances-0-gw8/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] Executing query SYSTEM SYNC REPLICA test_table_replicated on node3 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 Executing query SELECT COUNT() FROM test_table_replicated on node1 Stderr:time="2025-04-02T02:53:10Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestasyncconnecttomultipleips-gw0_default Creating Stderr: Network roottestasyncconnecttomultipleips-gw0_default Created Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Creating Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Creating Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Creating Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Created Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Created Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Created Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Starting Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Starting Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Starting Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Started Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Started Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Started Stderr:time="2025-04-02T02:53:11Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:53:11Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:10.0.0.3, port:2181, use_ssl:False Connecting to 10.0.0.3(10.0.0.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT COUNT() FROM test_table_replicated on node2 Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Stopping Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Stopped Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Removing Stderr: Container roottestattachpartitionwithlargedestination-gw8-node-1 Removed Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Stopping Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Stopping Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Stopping Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Stopped Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Removing Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Stopped Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Removing Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Stopped Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Removing Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo3-1 Removed Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo1-1 Removed Stderr: Container roottestattachpartitionwithlargedestination-gw8-zoo2-1 Removed Stderr: Network roottestattachpartitionwithlargedestination-gw8_default Removing Stderr: Network roottestattachpartitionwithlargedestination-gw8_default Removed Cleanup called Docker networks for project roottestattachpartitionwithlargedestination-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestattachpartitionwithlargedestination-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestattachpartitionwithlargedestination-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestattachpartitionwithlargedestination-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Connecting to 10.0.0.3(10.0.0.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Unstopped containers: {} No running containers for project: roottestattachpartitionwithlargedestination-gw8 Trying to prune unused networks... Executing query SELECT * FROM tbl on node1 Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_compatibility_merge_tree_settings/test.py::test_check_projections_compatibility Running tests in /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/test.py Cluster start called. is_up=False Docker networks for project roottestcompatibilitymergetreesettings-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcompatibilitymergetreesettings-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcompatibilitymergetreesettings-gw8 are DRIVER VOLUME NAME Cleanup called Connecting to 10.0.0.3(10.0.0.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr: Network roottestbufferprofile-gw9_default Creating Stderr: Network roottestbufferprofile-gw9_default Created Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Creating Stderr: Container roottestbufferprofile-gw9-node_default-1 Creating Stderr: Container roottestbufferprofile-gw9-node_default-1 Created Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Created Stderr: Container roottestbufferprofile-gw9-node_default-1 Starting Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Starting Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Started Stderr: Container roottestbufferprofile-gw9-node_default-1 Started ClickHouse instance created get_instance_ip instance_name=node_default http://localhost:None "GET /v1.46/containers/roottestbufferprofile-gw9-node_default-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node_default http://localhost:None "GET /v1.46/containers/roottestbufferprofile-gw9-node_default-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node_default, ip: 172.16.2.3... http://localhost:None "GET /v1.46/containers/roottestbufferprofile-gw9-node_default-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Docker networks for project roottestcompatibilitymergetreesettings-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcompatibilitymergetreesettings-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcompatibilitymergetreesettings-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcompatibilitymergetreesettings-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcompatibilitymergetreesettings-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query SELECT * FROM tbl on node2 Executing query SELECT COUNT() FROM test_table_replicated on node3 Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/database Setup logs dir /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/database Setup logs dir /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/configs/mt_settings.xml'] to /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/database Setup logs dir /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/.env --project-name roottestcompatibilitymergetreesettings-gw8 --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/docker-compose.yml pull] http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Connecting to 10.0.0.3(10.0.0.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: No route to host Connection dropped: socket connection error: No route to host Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_empty_replicated_table Executing query SELECT COUNT() FROM test_table_replicated on node4 http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Executing query TRUNCATE TABLE test_table_replicated on node1 Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Connecting to 10.0.0.3(10.0.0.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query TRUNCATE TABLE test_table_replicated on node2 http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Executing query TRUNCATE TABLE test_table_replicated on node3 http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query TRUNCATE TABLE test_table_replicated on node4 http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a2ae6c0ba4979ade12830c93558a86f6e376c11fd3b58cd5232735a9883a7af9/json HTTP/1.1" 200 None ClickHouse node_default started get_instance_ip instance_name=node_buffer_profile http://localhost:None "GET /v1.46/containers/roottestbufferprofile-gw9-node_buffer_profile-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node_buffer_profile http://localhost:None "GET /v1.46/containers/roottestbufferprofile-gw9-node_buffer_profile-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node_buffer_profile, ip: 172.16.2.2... http://localhost:None "GET /v1.46/containers/roottestbufferprofile-gw9-node_buffer_profile-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c22a53a9740682e861438c73428fd8ee34429f65e610448440169295427df1e0/json HTTP/1.1" 200 None ClickHouse node_buffer_profile started Executing query CREATE TABLE data (key Int) Engine=MergeTree() ORDER BY key PARTITION BY key % 2; CREATE TABLE buffer AS data Engine=Buffer(currentDatabase(), data, /* settings for manual flush only */ 1, /* num_layers */ 10e6, /* min_time, placeholder */ 10e6, /* max_time, placeholder */ 0, /* min_rows */ 10e6, /* max_rows */ 0, /* min_bytes */ 80e6 /* max_bytes */ ); INSERT INTO buffer SELECT * FROM numbers(100); on node_buffer_profile Command:[docker compose --env-file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/.env --project-name roottestalteronmixedtypecluster-gw7 --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/docker-compose.yml stop --timeout 20] [gw7] PASSED test_alter_on_mixed_type_cluster/test.py::test_alter_replicated_on_cluster Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query OPTIMIZE TABLE buffer on node_buffer_profile Connecting to 10.0.0.3(10.0.0.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused test_backup_restore_on_cluster/test.py::test_file_deduplication Executing query CREATE TABLE tbl ON CLUSTER 'cluster' ( vamqsNftDO Int32 ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}') ORDER BY tuple() SETTINGS min_bytes_for_wide_part=0 on node1 Executing query CREATE TABLE data (key Int) Engine=MergeTree() ORDER BY key PARTITION BY key % 2; CREATE TABLE buffer AS data Engine=Buffer(currentDatabase(), data, /* settings for manual flush only */ 1, /* num_layers */ 10e6, /* min_time, placeholder */ 10e6, /* max_time, placeholder */ 0, /* min_rows */ 10e6, /* max_rows */ 0, /* min_bytes */ 80e6 /* max_bytes */ ); INSERT INTO buffer SELECT * FROM numbers(100); on node_default [gw9] PASSED test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Executing query OPTIMIZE TABLE buffer on node_default Executing query CREATE TABLE tbl2 ON CLUSTER 'cluster' ( vamqsNftDO Int32 ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}-2') ORDER BY tuple() SETTINGS min_bytes_for_wide_part=0 on node1 Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/.env --project-name roottestconfigsubstitutions-gw4 --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node3/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_config_substitutions/_instances-0-gw4/node8/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/.env --project-name roottestbufferprofile-gw9 --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/docker-compose.yml --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/docker-compose.yml stop --timeout 20] [gw9] PASSED test_buffer_profile/test.py::test_default_profile Executing query INSERT INTO tbl VALUES (3556), (1177), (4004), (4264), (3729), (1438), (2158), (2684), (415), (1917) on node1 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl2 on node1 Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node5-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-node7-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-node4-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-node3-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Stopping Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node2-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Stopped Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Removing Stderr: Container roottestconfigsubstitutions-gw4-node8-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-node1-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-node6-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-zoo3-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-zoo1-1 Removed Stderr: Container roottestconfigsubstitutions-gw4-zoo2-1 Removed Stderr: Network roottestconfigsubstitutions-gw4_default Removing Stderr: Network roottestconfigsubstitutions-gw4_default Removed Cleanup called Docker networks for project roottestconfigsubstitutions-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestconfigsubstitutions-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestconfigsubstitutions-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestconfigsubstitutions-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestconfigsubstitutions-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_alternative_keeper_config/test.py::test_create_insert Running tests in /ClickHouse/tests/integration/test_alternative_keeper_config/test.py Cluster start called. is_up=False Docker networks for project roottestalternativekeeperconfig-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestalternativekeeperconfig-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestalternativekeeperconfig-gw4 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestalternativekeeperconfig-gw4 are NETWORK ID NAME DRIVER SCOPE Executing query BACKUP TABLE tbl, TABLE tbl2 ON CLUSTER 'cluster' TO Disk('backups', '7') on node1 Docker containers for project roottestalternativekeeperconfig-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestalternativekeeperconfig-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestalternativekeeperconfig-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestalternativekeeperconfig-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_alternative_keeper_config/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_alternative_keeper_config/configs/keeper_config.xml', '/ClickHouse/tests/integration/test_alternative_keeper_config/configs/enable_keeper1.xml'] to /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/database Setup logs dir /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_alternative_keeper_config/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_alternative_keeper_config/configs/zookeeper_config.xml', '/ClickHouse/tests/integration/test_alternative_keeper_config/configs/enable_keeper2.xml'] to /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/database Setup logs dir /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_alternative_keeper_config/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_alternative_keeper_config/configs/enable_keeper3.xml'] to /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/database Setup logs dir /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/.env --project-name roottestalternativekeeperconfig-gw4 --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/docker-compose.yml pull] Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster' on node1 Connecting to 10.0.0.3(10.0.0.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:10.0.0.2, port:2181, use_ssl:False Connecting to 10.0.0.2(10.0.0.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:10.0.0.4, port:2181, use_ssl:False Connecting to 10.0.0.4(10.0.0.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/.env --project-name roottestasyncconnecttomultipleips-gw0 --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_net.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/.env --project-name roottestasyncconnecttomultipleips-gw0 --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_net.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/docker-compose.yml up -d --no-recreate] run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'grep -a "Writing backup for file .*vamqsNftDO.bin .* (disk default)" /var/log/clickhouse-server/clickhouse-server.log | wc -l'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c grep -a "Writing backup for file .*vamqsNftDO.bin .* (disk default)" /var/log/clickhouse-server/clickhouse-server.log | wc -l] Stdout:1 run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'grep -a "Writing backup for file .*vamqsNftDO.bin .* (disk default)" /var/log/clickhouse-server/clickhouse-server.log | wc -l'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c grep -a "Writing backup for file .*vamqsNftDO.bin .* (disk default)" /var/log/clickhouse-server/clickhouse-server.log | wc -l] Stdout:0 run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'grep -a "Writing backup for file .*vamqsNftDO.bin .* skipped" /var/log/clickhouse-server/clickhouse-server.log | wc -l'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c grep -a "Writing backup for file .*vamqsNftDO.bin .* skipped" /var/log/clickhouse-server/clickhouse-server.log | wc -l] Stdout:3 run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'grep -a "Writing backup for file .*vamqsNftDO.bin .* skipped" /var/log/clickhouse-server/clickhouse-server.log | wc -l'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c grep -a "Writing backup for file .*vamqsNftDO.bin .* skipped" /var/log/clickhouse-server/clickhouse-server.log | wc -l] Stdout:0 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_file_deduplication Stderr: backward Pulling Stderr: upstream Pulling Stderr: 17d0386c2fff Pulling fs layer Stderr: c225dbfce283 Pulling fs layer Stderr: 2130ef613416 Pulling fs layer Stderr: 4f4fb700ef54 Pulling fs layer Stderr: 6a3fe6bcecc5 Pulling fs layer Stderr: old_node Pulling Stderr: 33bc578e5e14 Pulling fs layer Stderr: 6d0df3aa878d Pulling fs layer Stderr: upstream_node Pulling Stderr: 690edf6c102b Pulling fs layer Stderr: upstream_node Pulled Stderr: 17d0386c2fff Pulling fs layer Stderr: 87e28718f844 Pulling fs layer Stderr: c225dbfce283 Pulling fs layer Stderr: 6a3fe6bcecc5 Waiting Stderr: 4f4fb700ef54 Waiting Stderr: 2130ef613416 Pulling fs layer Stderr: 4f4fb700ef54 Pulling fs layer Stderr: 33bc578e5e14 Waiting Stderr: 6d0df3aa878d Waiting Stderr: 6a3fe6bcecc5 Pulling fs layer Stderr: 690edf6c102b Waiting Stderr: 33bc578e5e14 Pulling fs layer Stderr: 87e28718f844 Waiting Stderr: 6d0df3aa878d Pulling fs layer Stderr: 17d0386c2fff Downloading [> ] 296.5kB/27.51MB Stderr: c225dbfce283 Downloading [> ] 100.7kB/8.787MB Stderr: 690edf6c102b Pulling fs layer Stderr: 87e28718f844 Pulling fs layer Stderr: 2130ef613416 Downloading [> ] 526.6kB/258.8MB Stderr: c225dbfce283 Verifying Checksum Stderr: 17d0386c2fff Extracting [> ] 294.9kB/27.51MB Stderr: c225dbfce283 Download complete Stderr: 2130ef613416 Downloading [======> ] 33.8MB/258.8MB Stderr: c225dbfce283 Download complete Stderr: 6d0df3aa878d Download complete Stderr: 4f4fb700ef54 Downloading [==================================================>] 32B/32B Stderr: 33bc578e5e14 Download complete Stderr: 4f4fb700ef54 Verifying Checksum Stderr: 4f4fb700ef54 Download complete Stderr: 4f4fb700ef54 Download complete Stderr: 6a3fe6bcecc5 Download complete Stderr: 690edf6c102b Download complete Stderr: 6a3fe6bcecc5 Downloading [========> ] 3.646kB/22.46kB Stderr: 87e28718f844 Download complete Stderr: 17d0386c2fff Extracting [======> ] 3.539MB/27.51MB Stderr: 6a3fe6bcecc5 Download complete Stderr: 2130ef613416 Downloading [========> ] 43.8MB/258.8MB Stderr: 17d0386c2fff Extracting [==============> ] 7.963MB/27.51MB Stderr: 17d0386c2fff Downloading [===============> ] 8.552MB/27.51MB Stderr: 2130ef613416 Downloading [==========> ] 54.9MB/258.8MB Stderr: 2130ef613416 Downloading [=> ] 8.782MB/258.8MB Stderr: 17d0386c2fff Extracting [===================> ] 10.62MB/27.51MB Stderr: 2130ef613416 Downloading [=============> ] 67.64MB/258.8MB Stderr: 33bc578e5e14 Downloading [> ] 13.78kB/863.5kB Stderr: 17d0386c2fff Extracting [===============================> ] 17.4MB/27.51MB Stderr: 33bc578e5e14 Downloading [==================================================>] 863.5kB/863.5kB Stderr: 2130ef613416 Downloading [===============> ] 79.84MB/258.8MB Stderr: 17d0386c2fff Extracting [=========================================> ] 22.71MB/27.51MB Stderr: 33bc578e5e14 Download complete Stderr: 2130ef613416 Downloading [=================> ] 92.61MB/258.8MB Stderr: 6d0df3aa878d Downloading [==================================================>] 116B/116B Stderr: 17d0386c2fff Extracting [==========================================> ] 23.3MB/27.51MB Stderr: 2130ef613416 Downloading [====================> ] 105.4MB/258.8MB Stderr: 6d0df3aa878d Verifying Checksum Stderr: 17d0386c2fff Extracting [===============================================> ] 26.25MB/27.51MB Stderr: 2130ef613416 Downloading [======================> ] 117.6MB/258.8MB Stderr: 6d0df3aa878d Download complete Stderr: 17d0386c2fff Extracting [=================================================> ] 27.13MB/27.51MB Stderr: 690edf6c102b Downloading [==================================================>] 364B/364B Stderr: 2130ef613416 Downloading [=========================> ] 134.3MB/258.8MB Stderr: 17d0386c2fff Extracting [==================================================>] 27.51MB/27.51MB Stderr: 690edf6c102b Download complete Stderr: 2130ef613416 Downloading [================================> ] 165.9MB/258.8MB Stderr: 17d0386c2fff Pull complete Stderr: c225dbfce283 Extracting [> ] 98.3kB/8.787MB Stderr: 17d0386c2fff Downloading [==============================> ] 16.81MB/27.51MB Stderr: 2130ef613416 Downloading [======================================> ] 197.5MB/258.8MB Stderr: 2130ef613416 Downloading [===> ] 17.1MB/258.8MB Stderr: c225dbfce283 Extracting [===============> ] 2.753MB/8.787MB Stderr: 87e28718f844 Downloading [==================================================>] 2.936kB/2.936kB Stderr: 2130ef613416 Downloading [============================================> ] 230.8MB/258.8MB Stderr: c225dbfce283 Extracting [===============================> ] 5.603MB/8.787MB Stderr: 87e28718f844 Download complete Stderr: 2130ef613416 Download complete Stderr: c225dbfce283 Extracting [============================================> ] 7.766MB/8.787MB Stderr: upstream Pulled Stderr: c225dbfce283 Extracting [==============================================> ] 8.159MB/8.787MB Stderr: c225dbfce283 Extracting [===============================================> ] 8.356MB/8.787MB Stderr: 17d0386c2fff Downloading [==============================================> ] 25.66MB/27.51MB Stderr: c225dbfce283 Extracting [==================================================>] 8.787MB/8.787MB Stderr: 17d0386c2fff Verifying Checksum Stderr: c225dbfce283 Pull complete Stderr: 17d0386c2fff Download complete Stderr: 2130ef613416 Extracting [> ] 557.1kB/258.8MB Stderr: 2130ef613416 Extracting [=> ] 5.571MB/258.8MB Stderr: 2130ef613416 Downloading [====> ] 25.46MB/258.8MB Stderr: 2130ef613416 Extracting [===> ] 15.6MB/258.8MB Stderr: 17d0386c2fff Extracting [> ] 294.9kB/27.51MB Stderr: 2130ef613416 Extracting [====> ] 22.28MB/258.8MB Stderr: 2130ef613416 Extracting [=====> ] 28.97MB/258.8MB Stderr: 2130ef613416 Downloading [======> ] 33.8MB/258.8MB Stderr: 2130ef613416 Extracting [======> ] 36.21MB/258.8MB Stderr: 2130ef613416 Extracting [========> ] 43.45MB/258.8MB Stderr: 17d0386c2fff Extracting [======> ] 3.539MB/27.51MB Stderr: 2130ef613416 Extracting [=========> ] 49.02MB/258.8MB Stderr: 2130ef613416 Extracting [==========> ] 55.15MB/258.8MB Stderr: 2130ef613416 Downloading [========> ] 43.8MB/258.8MB Stderr: 2130ef613416 Extracting [===========> ] 60.16MB/258.8MB Stderr: 17d0386c2fff Extracting [==============> ] 7.963MB/27.51MB Stderr: 2130ef613416 Extracting [============> ] 66.29MB/258.8MB Stderr: 2130ef613416 Extracting [=============> ] 71.3MB/258.8MB Stderr: 2130ef613416 Downloading [==========> ] 54.9MB/258.8MB Stderr: 2130ef613416 Extracting [==============> ] 75.76MB/258.8MB Stderr: 17d0386c2fff Extracting [===================> ] 10.62MB/27.51MB Stderr: 2130ef613416 Extracting [===============> ] 80.22MB/258.8MB Stderr: 2130ef613416 Downloading [=============> ] 67.64MB/258.8MB Stderr: 2130ef613416 Extracting [=================> ] 88.57MB/258.8MB Stderr: 2130ef613416 Extracting [==================> ] 98.04MB/258.8MB Stderr: 17d0386c2fff Extracting [===============================> ] 17.4MB/27.51MB Stderr: 2130ef613416 Extracting [====================> ] 108.6MB/258.8MB Stderr: 2130ef613416 Downloading [===============> ] 79.84MB/258.8MB Stderr: 2130ef613416 Extracting [======================> ] 118.7MB/258.8MB Stderr: 2130ef613416 Extracting [========================> ] 128.7MB/258.8MB Stderr: 17d0386c2fff Extracting [=========================================> ] 22.71MB/27.51MB Stderr: 2130ef613416 Extracting [==========================> ] 137.6MB/258.8MB Stderr: 2130ef613416 Extracting [=============================> ] 151MB/258.8MB Stderr: 2130ef613416 Downloading [=================> ] 92.61MB/258.8MB Stderr: 2130ef613416 Extracting [==============================> ] 157.6MB/258.8MB Stderr: 2130ef613416 Extracting [===============================> ] 164.9MB/258.8MB Stderr: 2130ef613416 Extracting [=================================> ] 171.6MB/258.8MB Stderr: 17d0386c2fff Extracting [==========================================> ] 23.3MB/27.51MB Stderr: 2130ef613416 Extracting [===================================> ] 181.6MB/258.8MB Stderr: 2130ef613416 Extracting [=====================================> ] 191.6MB/258.8MB Stderr: 2130ef613416 Downloading [====================> ] 105.4MB/258.8MB Stderr: 2130ef613416 Extracting [======================================> ] 197.2MB/258.8MB Stderr: 17d0386c2fff Extracting [===============================================> ] 26.25MB/27.51MB Stderr: 2130ef613416 Extracting [========================================> ] 207.2MB/258.8MB Stderr: 2130ef613416 Extracting [=========================================> ] 216.1MB/258.8MB Stderr: 2130ef613416 Extracting [===========================================> ] 224.5MB/258.8MB Stderr: 2130ef613416 Downloading [======================> ] 117.6MB/258.8MB Stderr: 2130ef613416 Extracting [============================================> ] 232.8MB/258.8MB Stderr: 2130ef613416 Extracting [==============================================> ] 242.9MB/258.8MB Stderr: 17d0386c2fff Extracting [=================================================> ] 27.13MB/27.51MB Stderr: 2130ef613416 Extracting [================================================> ] 251.2MB/258.8MB Stderr: 2130ef613416 Extracting [=================================================> ] 256.8MB/258.8MB Stderr: 2130ef613416 Downloading [=========================> ] 134.3MB/258.8MB Stderr: 2130ef613416 Extracting [==================================================>] 258.8MB/258.8MB Stderr: 2130ef613416 Pull complete Stderr: 4f4fb700ef54 Extracting [==================================================>] 32B/32B Stderr: 17d0386c2fff Extracting [==================================================>] 27.51MB/27.51MB Stderr: 4f4fb700ef54 Extracting [==================================================>] 32B/32B Stderr: 4f4fb700ef54 Pull complete Stderr: 6a3fe6bcecc5 Extracting [==================================================>] 22.46kB/22.46kB Stderr: 2130ef613416 Downloading [================================> ] 165.9MB/258.8MB Stderr: 6a3fe6bcecc5 Extracting [==================================================>] 22.46kB/22.46kB Stderr: 6a3fe6bcecc5 Pull complete Stderr: 17d0386c2fff Pull complete Stderr: 33bc578e5e14 Extracting [=> ] 32.77kB/863.5kB Stderr: 33bc578e5e14 Extracting [==================================================>] 863.5kB/863.5kB Stderr: 33bc578e5e14 Extracting [==================================================>] 863.5kB/863.5kB Stderr: c225dbfce283 Extracting [> ] 98.3kB/8.787MB Stderr: 33bc578e5e14 Pull complete Stderr: 6d0df3aa878d Extracting [==================================================>] 116B/116B Stderr: 6d0df3aa878d Pull complete Stderr: 2130ef613416 Downloading [======================================> ] 197.5MB/258.8MB Stderr: 690edf6c102b Extracting [==================================================>] 364B/364B Stderr: c225dbfce283 Extracting [===============> ] 2.753MB/8.787MB Stderr: 690edf6c102b Extracting [==================================================>] 364B/364B Stderr: 690edf6c102b Pull complete Stderr: 87e28718f844 Extracting [==================================================>] 2.936kB/2.936kB Stderr: 2130ef613416 Downloading [============================================> ] 230.8MB/258.8MB Stderr: 87e28718f844 Extracting [==================================================>] 2.936kB/2.936kB Stderr: c225dbfce283 Extracting [===============================> ] 5.603MB/8.787MB Stderr: 87e28718f844 Pull complete Stderr: old_node Pulled Stderr: 2130ef613416 Verifying Checksum ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/.env --project-name roottestbackwardcompatibilityinsertprofileevents-gw6 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/docker-compose.yml up -d --no-recreate') Stderr: 2130ef613416 Download complete Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/.env --project-name roottestbackwardcompatibilityinsertprofileevents-gw6 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/docker-compose.yml up -d --no-recreate] Stderr: c225dbfce283 Extracting [============================================> ] 7.766MB/8.787MB Stderr: c225dbfce283 Extracting [==============================================> ] 8.159MB/8.787MB Stderr: c225dbfce283 Extracting [===============================================> ] 8.356MB/8.787MB Stderr: c225dbfce283 Extracting [==================================================>] 8.787MB/8.787MB Stderr: c225dbfce283 Pull complete Stderr: 2130ef613416 Extracting [> ] 557.1kB/258.8MB Stderr: 2130ef613416 Extracting [=> ] 5.571MB/258.8MB Stderr: 2130ef613416 Extracting [===> ] 15.6MB/258.8MB Stderr: 2130ef613416 Extracting [====> ] 22.28MB/258.8MB Stderr: 2130ef613416 Extracting [=====> ] 28.97MB/258.8MB Stderr: 2130ef613416 Extracting [======> ] 36.21MB/258.8MB Stderr: 2130ef613416 Extracting [========> ] 43.45MB/258.8MB Stderr: 2130ef613416 Extracting [=========> ] 49.02MB/258.8MB Stderr: 2130ef613416 Extracting [==========> ] 55.15MB/258.8MB Stderr: 2130ef613416 Extracting [===========> ] 60.16MB/258.8MB Stderr: 2130ef613416 Extracting [============> ] 66.29MB/258.8MB Stderr: 2130ef613416 Extracting [=============> ] 71.3MB/258.8MB Stderr: 2130ef613416 Extracting [==============> ] 75.76MB/258.8MB Stderr: 2130ef613416 Extracting [===============> ] 80.22MB/258.8MB Stderr: 2130ef613416 Extracting [=================> ] 88.57MB/258.8MB Stderr: 2130ef613416 Extracting [==================> ] 98.04MB/258.8MB Stderr: 2130ef613416 Extracting [====================> ] 108.6MB/258.8MB Stderr: 2130ef613416 Extracting [======================> ] 118.7MB/258.8MB Stderr: 2130ef613416 Extracting [========================> ] 128.7MB/258.8MB Stderr: 2130ef613416 Extracting [==========================> ] 137.6MB/258.8MB Stderr: 2130ef613416 Extracting [=============================> ] 151MB/258.8MB Stderr: 2130ef613416 Extracting [==============================> ] 157.6MB/258.8MB Stderr: 2130ef613416 Extracting [===============================> ] 164.9MB/258.8MB Stderr: 2130ef613416 Extracting [=================================> ] 171.6MB/258.8MB Stderr: 2130ef613416 Extracting [===================================> ] 181.6MB/258.8MB Stderr: 2130ef613416 Extracting [=====================================> ] 191.6MB/258.8MB Stderr: 2130ef613416 Extracting [======================================> ] 197.2MB/258.8MB Stderr: 2130ef613416 Extracting [========================================> ] 207.2MB/258.8MB Stderr: 2130ef613416 Extracting [=========================================> ] 216.1MB/258.8MB Stderr: 2130ef613416 Extracting [===========================================> ] 224.5MB/258.8MB Stderr: 2130ef613416 Extracting [============================================> ] 232.8MB/258.8MB Stderr: 2130ef613416 Extracting [==============================================> ] 242.9MB/258.8MB Stderr: 2130ef613416 Extracting [================================================> ] 251.2MB/258.8MB Stderr: 2130ef613416 Extracting [=================================================> ] 256.8MB/258.8MB Stderr: 2130ef613416 Extracting [==================================================>] 258.8MB/258.8MB Stderr: 2130ef613416 Pull complete Stderr: 4f4fb700ef54 Extracting [==================================================>] 32B/32B Stderr: 4f4fb700ef54 Extracting [==================================================>] 32B/32B Stderr: 4f4fb700ef54 Pull complete Stderr: 6a3fe6bcecc5 Extracting [==================================================>] 22.46kB/22.46kB Stderr: 6a3fe6bcecc5 Extracting [==================================================>] 22.46kB/22.46kB Stderr: 6a3fe6bcecc5 Pull complete Stderr: 33bc578e5e14 Extracting [=> ] 32.77kB/863.5kB Stderr: 33bc578e5e14 Extracting [==================================================>] 863.5kB/863.5kB Stderr: 33bc578e5e14 Extracting [==================================================>] 863.5kB/863.5kB Stderr: 33bc578e5e14 Pull complete Stderr: 6d0df3aa878d Extracting [==================================================>] 116B/116B Stderr: 6d0df3aa878d Extracting [==================================================>] 116B/116B Stderr: 6d0df3aa878d Pull complete Stderr: 690edf6c102b Extracting [==================================================>] 364B/364B Stderr: 690edf6c102b Extracting [==================================================>] 364B/364B Stderr: 690edf6c102b Pull complete Stderr: 87e28718f844 Extracting [==================================================>] 2.936kB/2.936kB Stderr: 87e28718f844 Extracting [==================================================>] 2.936kB/2.936kB Stderr: 87e28718f844 Pull complete Stderr: backward Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/.env --project-name roottestbackwardcompatibilityfunctions-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/.env --project-name roottestbackwardcompatibilityfunctions-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/docker-compose.yml up -d --no-recreate] Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Running Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Running Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Running Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Creating Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Creating Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Created Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Created Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Starting Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Starting Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Started Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 10.5.95.11... http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None Stderr: Network roottestbackwardcompatibilityinsertprofileevents-gw6_default Creating Stderr: Network roottestbackwardcompatibilityinsertprofileevents-gw6_default Created Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Creating Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Creating Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Created Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Created Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Starting Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Starting Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Started Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Started ClickHouse instance created get_instance_ip instance_name=upstream_node http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=upstream_node http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in upstream_node, ip: 172.16.1.3... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1/json HTTP/1.1" 200 None Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None Stderr: Network roottestbackwardcompatibilityfunctions-gw5_default Creating Stderr: Network roottestbackwardcompatibilityfunctions-gw5_default Created Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Creating Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Creating Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Created Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Created Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Starting Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Starting Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Started Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Started ClickHouse instance created get_instance_ip instance_name=upstream http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityfunctions-gw5-upstream-1/json HTTP/1.1" 200 None get_instance_ip instance_name=upstream http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityfunctions-gw5-upstream-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in upstream, ip: 172.16.4.2... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityfunctions-gw5-upstream-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Stopping Stderr: Container roottestbufferprofile-gw9-node_default-1 Stopping Stderr: Container roottestbufferprofile-gw9-node_default-1 Stopped Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] http://localhost:None "GET /v1.46/containers/e54909d87d7c78a9302bc537729682000c30b29ac20c9cec0a00fc6602e00f87/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 Command:[docker compose --env-file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/.env --project-name roottestbufferprofile-gw9 --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_default/docker-compose.yml --file /ClickHouse/tests/integration/test_buffer_profile/_instances-0-gw9/node_buffer_profile/docker-compose.yml down --volumes] http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-node2-1/json HTTP/1.1" 200 None Connection dropped: socket connection error: Connection refused get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 10.5.95.12... http://localhost:None "GET /v1.46/containers/roottestasyncconnecttomultipleips-gw0-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9b7ad14602bc449b3c68074947274cb479c9e8f068df402b5b9c6fffcd9a8d11/json HTTP/1.1" 200 None ClickHouse node2 started Executing query CREATE TABLE test(t Date, label UInt8) ENGINE = MergeTree PARTITION BY t ORDER BY label; on node1 http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None Executing query CREATE TABLE tbl (`x` UInt8) ENGINE = MergeTree ORDER BY x on node1 test_backup_restore_on_cluster/test.py::test_get_error_from_other_host http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Stopping Stderr: Container roottestbufferprofile-gw9-node_default-1 Stopping Stderr: Container roottestbufferprofile-gw9-node_default-1 Stopped Stderr: Container roottestbufferprofile-gw9-node_default-1 Removing Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Stopped Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Removing Stderr: Container roottestbufferprofile-gw9-node_default-1 Removed Stderr: Container roottestbufferprofile-gw9-node_buffer_profile-1 Removed Stderr: Network roottestbufferprofile-gw9_default Removing Stderr: Network roottestbufferprofile-gw9_default Removed Cleanup called http://localhost:None "GET /v1.46/containers/a3393b76c52907ffac0060a4b1cab944add8324e6a0d3516b90bbcfc08576dc9/json HTTP/1.1" 200 None ClickHouse upstream_node started get_instance_ip instance_name=old_node http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=old_node http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in old_node, ip: 172.16.1.2... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d96d1b418f17455373c0a3118de201bf55b45f943afb7b0e5414faa7abcd80c5/json HTTP/1.1" 200 None ClickHouse old_node started Executing query INSERT INTO FUNCTION null('foo String') VALUES ('foo')('bar') on upstream_node Docker networks for project roottestbufferprofile-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbufferprofile-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbufferprofile-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbufferprofile-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None Unstopped containers: {} No running containers for project: roottestbufferprofile-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_backward_compatibility/test_convert_ordinary.py::test_convert_ordinary_to_atomic Running tests in /ClickHouse/tests/integration/test_backward_compatibility/test_convert_ordinary.py Cluster start called. is_up=False Docker networks for project roottestbackwardcompatibilityconvertordinary-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityconvertordinary-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES http://localhost:None "GET /v1.46/containers/5ef41fe36252a2ddb088d21fdd88b4036c9f69da43cfc76feffd566ea9116575/json HTTP/1.1" 200 None ClickHouse upstream started get_instance_ip instance_name=backward http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityfunctions-gw5-backward-1/json HTTP/1.1" 200 None get_instance_ip instance_name=backward http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityfunctions-gw5-backward-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in backward, ip: 172.16.4.3... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityfunctions-gw5-backward-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/a1d5098bad1ef7a55a5c76769dadd0995bdd3245b2c822cc4ce99303af3bd6f3/json HTTP/1.1" 200 None Executing query INSERT INTO test SELECT toDate('2022-12-28'), 1; on node1 ClickHouse backward started Executing query SELECT if(NOT empty(alias_to), alias_to, name) FROM system.functions WHERE is_aggregate = 1 on backward Executing query INSERT INTO tbl VALUES (3) on node1 Docker volumes for project roottestbackwardcompatibilityconvertordinary-gw9 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbackwardcompatibilityconvertordinary-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityconvertordinary-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query INSERT INTO FUNCTION null('foo String') VALUES ('foo')('bar') on upstream_node Docker volumes for project roottestbackwardcompatibilityconvertordinary-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityconvertordinary-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityconvertordinary-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/.env --project-name roottestbackwardcompatibilityconvertordinary-gw9 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml pull] Executing query SELECT count(*) FROM test on node1 Got 163 aggregate functions Checking exponentialTimeDecayedCount Executing query select hex(initializeAggregation('exponentialTimeDecayedCountState', 'foo')) on backward Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '8') on node1 [gw6] PASSED test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible Executing query INSERT INTO FUNCTION null('foo String') VALUES ('foo')('bar') on old_node run container_id:roottestasyncconnecttomultipleips-gw0-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '2001:3984:3989::1:1118 node1' >> /etc/hosts"] Command:[docker exec roottestasyncconnecttomultipleips-gw0-node2-1 bash -c echo '2001:3984:3989::1:1118 node1' >> /etc/hosts] Skipping exponentialTimeDecayedCount Checking exponentialTimeDecayedMax Executing query select hex(initializeAggregation('exponentialTimeDecayedMaxState', 'foo')) on backward run container_id:roottestasyncconnecttomultipleips-gw0-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '10.5.95.11 node1' >> /etc/hosts"] Command:[docker exec roottestasyncconnecttomultipleips-gw0-node2-1 bash -c echo '10.5.95.11 node1' >> /etc/hosts] Executing query INSERT INTO FUNCTION null('foo String') VALUES ('foo')('bar') on old_node Executing query SELECT count(*) from test on node1 Skipping exponentialTimeDecayedMax Checking lagInFrame Executing query select hex(initializeAggregation('lagInFrameState', 'foo')) on backward Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_get_error_from_other_host Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/.env --project-name roottestbackwardcompatibilityinsertprofileevents-gw6 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/docker-compose.yml stop --timeout 20] [gw6] PASSED test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible Executing query SYSTEM DROP DNS CACHE on node2 Connection dropped: socket connection error: Connection refused Skipping lagInFrame Checking dense_rank Executing query select hex(initializeAggregation('dense_rankState', 'foo')) on backward Executing query SYSTEM DROP DNS CACHE on node1 Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Skipping dense_rank Checking rank Executing query select hex(initializeAggregation('rankState', 'foo')) on backward Command:[bash -c [ -f /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/.env --project-name roottestalteronmixedtypecluster-gw7 --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_alter_on_mixed_type_cluster/_instances-0-gw7/node4/docker-compose.yml down --volumes] Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Skipping rank Checking exponentialMovingAverage Executing query select hex(initializeAggregation('exponentialMovingAverageState', 'foo')) on backward Executing query SELECT count(*) FROM remote('node1', default.test) limit 1; on node2 Skipping exponentialMovingAverage Checking sparkbar Executing query select hex(initializeAggregation('sparkbarState', 'foo')) on backward Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Skipping sparkbar Checking singleValueOrNull Executing query select hex(initializeAggregation('singleValueOrNullState', 'foo')) on backward Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Removing Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Removing Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Removing Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Removing Stderr: Container roottestalteronmixedtypecluster-gw7-node1-1 Removed Stderr: Container roottestalteronmixedtypecluster-gw7-node4-1 Removed Stderr: Container roottestalteronmixedtypecluster-gw7-node3-1 Removed Stderr: Container roottestalteronmixedtypecluster-gw7-node2-1 Removed Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Stopping Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Removing Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Removing Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Stopped Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Removing Stderr: Container roottestalteronmixedtypecluster-gw7-zoo3-1 Removed Stderr: Container roottestalteronmixedtypecluster-gw7-zoo2-1 Removed Stderr: Container roottestalteronmixedtypecluster-gw7-zoo1-1 Removed Stderr: Network roottestalteronmixedtypecluster-gw7_default Removing Stderr: Network roottestalteronmixedtypecluster-gw7_default Removed Cleanup called Docker networks for project roottestalteronmixedtypecluster-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestalteronmixedtypecluster-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestalteronmixedtypecluster-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestalteronmixedtypecluster-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Unstopped containers: {} No running containers for project: roottestalteronmixedtypecluster-gw7 Trying to prune unused networks... Executing query select hex(initializeAggregation('singleValueOrNullState', 'foo')) on upstream Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_aggregation_memory_efficient/test.py::test_remote Running tests in /ClickHouse/tests/integration/test_aggregation_memory_efficient/test.py Cluster start called. is_up=False Docker networks for project roottestaggregationmemoryefficient-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestaggregationmemoryefficient-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestaggregationmemoryefficient-gw7 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestaggregationmemoryefficient-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestaggregationmemoryefficient-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestaggregationmemoryefficient-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestaggregationmemoryefficient-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestaggregationmemoryefficient-gw7 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] OK singleValueOrNull Checking meanZTest Executing query select hex(initializeAggregation('meanZTestState', 'foo')) on backward Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/database Setup logs dir /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/database Setup logs dir /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/.env --project-name roottestaggregationmemoryefficient-gw7 --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/docker-compose.yml pull] Skipping meanZTest Checking analysisOfVariance Executing query select hex(initializeAggregation('analysisOfVarianceState', 'foo')) on backward test_backup_restore_on_cluster/test.py::test_keeper_value_max_size Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Skipping analysisOfVariance Checking studentTTest Executing query select hex(initializeAggregation('studentTTestState', 'foo')) on backward Skipping studentTTest Checking rankCorr Executing query select hex(initializeAggregation('rankCorrState', 'foo')) on backward Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query INSERT INTO tbl VALUES (111) on node1 Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Skipping rankCorr Checking aggThrow Executing query select hex(initializeAggregation('aggThrowState', 'foo')) on backward Skipping aggThrow Checking categoricalInformationValue Executing query select hex(initializeAggregation('categoricalInformationValueState', 'foo')) on backward Executing query INSERT INTO tbl VALUES (222) on node2 Skipping categoricalInformationValue Checking groupArrayMovingAvg Executing query select hex(initializeAggregation('groupArrayMovingAvgState', 'foo')) on backward Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 Skipping groupArrayMovingAvg Checking groupArrayMovingSum Executing query select hex(initializeAggregation('groupArrayMovingSumState', 'foo')) on backward Executing query SYSTEM STOP REPLICATED SENDS ON CLUSTER 'cluster' tbl on node1 Skipping groupArrayMovingSum Checking simpleLinearRegression Executing query select hex(initializeAggregation('simpleLinearRegressionState', 'foo')) on backward Stderr: node1 Skipped - Image is already being pulled by node2 Stderr: node2 Pulling Stderr: node2 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/.env --project-name roottestaggregationmemoryefficient-gw7 --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/.env --project-name roottestaggregationmemoryefficient-gw7 --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/docker-compose.yml up -d --no-recreate] Stderr: node2 Skipped - Image is already being pulled by zoo1 Stderr: node3 Skipped - Image is already being pulled by zoo1 Stderr: node1 Skipped - Image is already being pulled by zoo1 Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: zoo1 Pulling Stderr: zoo1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper1/log', '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper1/config', '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper1/coordination', '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper2/log', '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper2/config', '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper2/coordination', '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper3/log', '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper3/config', '/ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/keeper3/coordination'] Command:[docker compose --project-name roottestcompatibilitymergetreesettings-gw8 --env-file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Executing query INSERT INTO tbl VALUES (333) on node1 Skipping simpleLinearRegression Checking entropy Executing query select hex(initializeAggregation('entropyState', 'foo')) on backward Stderr: node2 Skipped - Image is already being pulled by node1 Stderr: node3 Skipped - Image is already being pulled by node1 Stderr: node1 Pulling Stderr: node1 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/.env --project-name roottestalternativekeeperconfig-gw4 --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/.env --project-name roottestalternativekeeperconfig-gw4 --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/docker-compose.yml up -d --no-recreate] Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: node Pulling Stderr: zoo1 Pulling Stderr: zoo1 Pulled Stderr: node Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper1/log', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper1/config', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper1/coordination', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper2/log', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper2/config', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper2/coordination', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper3/log', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper3/config', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/keeper3/coordination'] Command:[docker compose --project-name roottestbackwardcompatibilityconvertordinary-gw9 --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Executing query INSERT INTO tbl VALUES (444) on node2 Executing query select hex(initializeAggregation('entropyState', 'foo')) on upstream Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Stopping Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Stopping Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Stopped Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] OK entropy Checking histogram Executing query select hex(initializeAggregation('histogramState', 'foo')) on backward Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '9') on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/.env --project-name roottestbackwardcompatibilityinsertprofileevents-gw6 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/upstream_node/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-insert_profile_events-0-gw6/old_node/docker-compose.yml down --volumes] Stderr: Network roottestaggregationmemoryefficient-gw7_default Creating Stderr: Network roottestaggregationmemoryefficient-gw7_default Created Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Creating Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Creating Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Created Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Created Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Starting Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Starting Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Started Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestaggregationmemoryefficient-gw7-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestaggregationmemoryefficient-gw7-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.2.2... http://localhost:None "GET /v1.46/containers/roottestaggregationmemoryefficient-gw7-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Skipping histogram Checking mannWhitneyUTest Executing query select hex(initializeAggregation('mannWhitneyUTestState', 'foo')) on backward Command:[docker compose --env-file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/.env --project-name roottestasyncconnecttomultipleips-gw0 --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_net.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/docker-compose.yml stop --timeout 20] [gw0] PASSED test_async_connect_to_multiple_ips/test.py::test http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Skipping mannWhitneyUTest Checking maxIntersections Executing query select hex(initializeAggregation('maxIntersectionsState', 'foo')) on backward Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Stopping Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Stopping Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Stopped Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Removing Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Stopped Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Removing Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-upstream_node-1 Removed Stderr: Container roottestbackwardcompatibilityinsertprofileevents-gw6-old_node-1 Removed Stderr: Network roottestbackwardcompatibilityinsertprofileevents-gw6_default Removing Stderr: Network roottestbackwardcompatibilityinsertprofileevents-gw6_default Removed Cleanup called Docker networks for project roottestbackwardcompatibilityinsertprofileevents-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityinsertprofileevents-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Docker volumes for project roottestbackwardcompatibilityinsertprofileevents-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityinsertprofileevents-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityinsertprofileevents-gw6 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Running tests in /ClickHouse/tests/integration/test_broken_part_during_merge/test.py test_broken_part_during_merge/test.py::test_merge_and_part_corruption Cluster start called. is_up=False http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Docker networks for project roottestbrokenpartduringmerge-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbrokenpartduringmerge-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbrokenpartduringmerge-gw6 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbrokenpartduringmerge-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbrokenpartduringmerge-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbrokenpartduringmerge-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbrokenpartduringmerge-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Unstopped containers: {} No running containers for project: roottestbrokenpartduringmerge-gw6 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/database Setup logs dir /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/.env --project-name roottestbrokenpartduringmerge-gw6 --file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml pull] http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Stderr:time="2025-04-02T02:53:22Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestcompatibilitymergetreesettings-gw8_default Creating Stderr: Network roottestcompatibilitymergetreesettings-gw8_default Created Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Creating Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Creating Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Creating Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Created Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Created Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Created Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Starting Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Starting Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Starting Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Started Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Started Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Started Stderr:time="2025-04-02T02:53:24Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:53:24Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.5.2, port:2181, use_ssl:False Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Skipping maxIntersections Checking groupBitmapXor Executing query select hex(initializeAggregation('groupBitmapXorState', 'foo')) on backward Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Stderr: Network roottestalternativekeeperconfig-gw4_default Creating Stderr: Network roottestalternativekeeperconfig-gw4_default Created Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Creating Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Creating Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Creating Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Created Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Created Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Created Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Starting Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Starting Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Starting Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Started Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Started Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.6.3... Stderr:time="2025-04-02T02:53:22Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestbackwardcompatibilityconvertordinary-gw9_default Creating Stderr: Network roottestbackwardcompatibilityconvertordinary-gw9_default Created Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Creating Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Creating Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Creating Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Created Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Created Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Created Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Starting Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Starting Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Starting Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Started Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Started http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node1-1/json HTTP/1.1" 200 None Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Started Stderr:time="2025-04-02T02:53:24Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:53:24Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.7.3, port:2181, use_ssl:False Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Skipping groupBitmapXor Checking groupBitmapAnd Executing query select hex(initializeAggregation('groupBitmapAndState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Skipping groupBitmapAnd Checking cramersVBiasCorrected Executing query select hex(initializeAggregation('cramersVBiasCorrectedState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '9') on node1 Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Skipping cramersVBiasCorrected Checking contingency Executing query select hex(initializeAggregation('contingencyState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5d0a16cfef9c467e4f16b1b589fbaad33ccb11b36dc2178156d9279592922825/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestaggregationmemoryefficient-gw7-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestaggregationmemoryefficient-gw7-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.2.3... http://localhost:None "GET /v1.46/containers/roottestaggregationmemoryefficient-gw7-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ddd94215d8b974ba86d57a3b85abb6a7598448bbc2537bf14f3b69761262ca0d/json HTTP/1.1" 200 None ClickHouse node2 started Executing query create table da_memory_efficient_shard(A Int64, B Int64) Engine=MergeTree order by A partition by B % 2; on node1 Skipping contingency Checking skewPop Executing query select hex(initializeAggregation('skewPopState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 Executing query create table da_memory_efficient_shard(A Int64, B Int64) Engine=MergeTree order by A partition by B % 2; on node2 Skipping skewPop Checking groupBitXor Executing query select hex(initializeAggregation('groupBitXorState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query SELECT * FROM tbl ORDER BY x on node1 http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query insert into da_memory_efficient_shard select number, number from numbers(100000); on node1 Skipping groupBitXor Checking groupBitOr Executing query select hex(initializeAggregation('groupBitOrState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query SELECT * FROM tbl ORDER BY x on node2 Skipping groupBitOr Checking exponentialTimeDecayedSum Executing query select hex(initializeAggregation('exponentialTimeDecayedSumState', 'foo')) on backward Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query insert into da_memory_efficient_shard select number + 100000, number from numbers(100000); on node2 Skipping exponentialTimeDecayedSum Checking topKWeighted Executing query select hex(initializeAggregation('topKWeightedState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_keeper_value_max_size http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Skipping topKWeighted Checking stochasticLinearRegression Executing query select hex(initializeAggregation('stochasticLinearRegressionState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query set distributed_aggregation_memory_efficient = 1, group_by_two_level_threshold = 1, group_by_two_level_threshold_bytes=1 on node1 http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Skipping stochasticLinearRegression Checking corr Executing query select hex(initializeAggregation('corrState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query select sum(a) from (SELECT B, uniqExact(A) a FROM remote('node{1,2}', default.da_memory_efficient_shard) GROUP BY B) on node1 http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Skipping corr Checking uniqCombined64 Executing query select hex(initializeAggregation('uniqCombined64State', 'foo')) on backward Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query select hex(initializeAggregation('uniqCombined64State', 'foo')) on upstream http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query set distributed_aggregation_memory_efficient = 0 on node1 http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None OK uniqCombined64 Checking intervalLengthSum Executing query select hex(initializeAggregation('intervalLengthSumState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query select sum(a) from (SELECT B, uniqExact(A) a FROM remote('node{1,2}', default.da_memory_efficient_shard) GROUP BY B) on node1 Skipping intervalLengthSum Checking nothing Executing query select hex(initializeAggregation('nothingState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None Executing query select hex(initializeAggregation('nothingState', 'foo')) on upstream test_backup_restore_on_cluster/test.py::test_mutation Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8, y String) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY tuple() on node1 http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ed947fb8324b34b20489edc56e1e8c3b0a6753f340a5f1fb9966198420c8efbf/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.6.2... http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b5ce8662062d805d04d0c5ca7da27458820a01bfd1418ec6cdcd34c03aa6d3a2/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node3-1/json HTTP/1.1" 200 None Executing query set distributed_aggregation_memory_efficient = 1, group_by_two_level_threshold = 1, group_by_two_level_threshold_bytes=1 on node1 get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.6.4... http://localhost:None "GET /v1.46/containers/roottestalternativekeeperconfig-gw4-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fa9d49dc73958fb65016088f2412b4cc77814b209d6abd86daa822e49b896be4/json HTTP/1.1" 200 None ClickHouse node3 started Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'test_cluster' SYNC on node1 Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost OK nothing Checking uniqCombined Executing query select hex(initializeAggregation('uniqCombinedState', 'foo')) on backward Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.7.4, port:2181, use_ssl:False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT fullHostName() AS h, uniqExact(A) AS a FROM remote('node{1,2}', default.da_memory_efficient_shard) GROUP BY h ORDER BY h; on node1 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.7.2, port:2181, use_ssl:False Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query INSERT INTO tbl SELECT number, toString(number) FROM numbers(5) on node1 Executing query CREATE TABLE tbl ON CLUSTER 'test_cluster' ( id Int64, str String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}') ORDER BY id on node1 Executing query select hex(initializeAggregation('uniqCombinedState', 'foo')) on upstream Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/.env --project-name roottestbackwardcompatibilityconvertordinary-gw9 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/.env --project-name roottestbackwardcompatibilityconvertordinary-gw9 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate] Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query INSERT INTO tbl SELECT number, toString(number) FROM numbers(5, 5) on node2 Executing query set distributed_aggregation_memory_efficient = 0 on node1 OK uniqCombined Checking sumMapFiltered Executing query select hex(initializeAggregation('sumMapFilteredState', 'foo')) on backward Executing query SELECT fullHostName() AS h, uniqExact(A) AS a FROM remote('node{1,2}', default.da_memory_efficient_shard) GROUP BY h ORDER BY h; on node1 Executing query INSERT INTO tbl SELECT number, toString(number) FROM numbers(10, 5) on node1 Executing query INSERT INTO tbl VALUES (1, 'str1') on node1 Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Running Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Running Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Running Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Creating Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Created Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Starting Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.7.5... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/2a23506c87f89e48327d4bd98b5aef48da910df7b91a20e1bc44ccdebd5d3350/json HTTP/1.1" 200 None Skipping sumMapFiltered Checking minMappedArrays Executing query select hex(initializeAggregation('minMappedArraysState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/2a23506c87f89e48327d4bd98b5aef48da910df7b91a20e1bc44ccdebd5d3350/json HTTP/1.1" 200 None Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query ALTER TABLE tbl UPDATE x=x+1 WHERE 1 on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/.env --project-name roottestasyncconnecttomultipleips-gw0 --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_net.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_async_connect_to_multiple_ips/_instances-0-gw0/node2/docker-compose.yml down --volumes] Executing query INSERT INTO tbl VALUES (1, 'str1') on node2 http://localhost:None "GET /v1.46/containers/2a23506c87f89e48327d4bd98b5aef48da910df7b91a20e1bc44ccdebd5d3350/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/.env --project-name roottestaggregationmemoryefficient-gw7 --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/docker-compose.yml stop --timeout 20] [gw7] PASSED test_aggregation_memory_efficient/test.py::test_remote Skipping minMappedArrays Checking sumMappedArrays Executing query select hex(initializeAggregation('sumMappedArraysState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/2a23506c87f89e48327d4bd98b5aef48da910df7b91a20e1bc44ccdebd5d3350/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE DATABASE default2 ENGINE=Ordinary on node Executing query ALTER TABLE tbl UPDATE x=x+1+sleep(3) WHERE 1 on node1 Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query INSERT INTO tbl VALUES (2, 'str2') on node3 Skipping sumMappedArrays Checking quantiles Executing query select hex(initializeAggregation('quantilesState', 'foo')) on backward Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.5.4, port:2181, use_ssl:False Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Executing query CREATE TABLE default2.t(date Date, id UInt32) ENGINE = MergeTree PARTITION BY toYYYYMM(date) ORDER BY id on node Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query ALTER TABLE tbl UPDATE x=x+1+sleep(3) WHERE 1 on node1 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.5.3, port:2181, use_ssl:False Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Skipping quantiles Checking sum Executing query select hex(initializeAggregation('sumState', 'foo')) on backward Executing query SYSTEM SYNC REPLICA ON CLUSTER 'test_cluster' tbl on node1 Executing query INSERT INTO default2.t VALUES (today(), 1) on node Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/.env --project-name roottestcompatibilitymergetreesettings-gw8 --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/.env --project-name roottestcompatibilitymergetreesettings-gw8 --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/docker-compose.yml up -d --no-recreate] Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '10') on node1 Skipping sum Checking sumKahan Executing query select hex(initializeAggregation('sumKahanState', 'foo')) on backward Executing query INSERT INTO default2.t SELECT number % 1000, number FROM system.numbers LIMIT 1000000 on node Executing query SELECT * FROM tbl ORDER BY id on node1 Skipping sumKahan Checking corrMatrix Executing query select hex(initializeAggregation('corrMatrixState', 'foo')) on backward Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Removing Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Removing Stderr: Container roottestasyncconnecttomultipleips-gw0-node1-1 Removed Stderr: Container roottestasyncconnecttomultipleips-gw0-node2-1 Removed Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Stopping Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Removing Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Removing Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Stopped Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Removing Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo3-1 Removed Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo2-1 Removed Stderr: Container roottestasyncconnecttomultipleips-gw0-zoo1-1 Removed Stderr: Network roottestasyncconnecttomultipleips-gw0_default Removing Stderr: Network roottestasyncconnecttomultipleips-gw0_default Removed Cleanup called Docker networks for project roottestasyncconnecttomultipleips-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestasyncconnecttomultipleips-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestasyncconnecttomultipleips-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestasyncconnecttomultipleips-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestasyncconnecttomultipleips-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query SELECT count() FROM default2.t on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 ENV HOSTNAME 6f72c710c5a6 ENV SHLVL 0 ENV HOME /root ENV OLDPWD / ENV DOCKER_HELPER_TAG 5dc43a6382f0 ENV PYTHONUNBUFFERED 1 ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e ENV UBSAN_OPTIONS print_stacktrace=1 ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_access_for_functions/test.py::test_access_rights_for_function test_access_for_functions/test.py::test_ignore_obsolete_grant_on_database test_aggregation_memory_efficient/test.py::test_remote test_allowed_client_hosts/test.py::test_allowed_host test_allowed_client_hosts/test.py::test_denied_host test_alter_codec/test.py::test_alter_codec_index test_alter_codec/test.py::test_alter_codec_pk test_alter_on_mixed_type_cluster/test.py::test_alter_on_cluter_non_replicated test_alter_on_mixed_type_cluster/test.py::test_alter_replicated_on_cluster test_alternative_keeper_config/test.py::test_create_insert test_async_connect_to_multiple_ips/test.py::test test_async_metrics_in_cgroup/test.py::test_normalized_user_cpu test_async_metrics_in_cgroup/test.py::test_system_wide_metrics test_async_metrics_in_cgroup/test.py::test_user_cpu_accounting 'test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Atomic]' 'test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Ordinary]' test_attach_table_normalizer/test.py::test_attach_substr test_attach_table_normalizer/test.py::test_attach_substr_restart test_authentication/test.py::test_authentication_fail test_authentication/test.py::test_authentication_pass test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication test_backup_restore/test.py::test_attach_partition test_backup_restore/test.py::test_replace_partition test_backup_restore/test.py::test_restore test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True]' 'test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True]' test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes test_backup_restore_on_cluster/test.py::test_empty_replicated_table test_backup_restore_on_cluster/test.py::test_file_deduplication test_backup_restore_on_cluster/test.py::test_get_error_from_other_host test_backup_restore_on_cluster/test.py::test_keeper_value_max_size test_backup_restore_on_cluster/test.py::test_mutation test_backup_restore_on_cluster/test.py::test_projection test_backup_restore_on_cluster/test.py::test_replicated_database test_backup_restore_on_cluster/test.py::test_replicated_database_async test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 'test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[database]' 'test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[uuid]' test_backup_restore_on_cluster/test.py::test_replicated_table test_backup_restore_on_cluster/test.py::test_replicated_table_after_alters test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath test_backup_restore_on_cluster/test.py::test_required_privileges test_backup_restore_on_cluster/test.py::test_shutdown_waits_for_backup test_backup_restore_on_cluster/test.py::test_system_functions test_backup_restore_on_cluster/test.py::test_system_users test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty test_backup_restore_on_cluster/test.py::test_tables_dependency test_backward_compatibility/test_convert_ordinary.py::test_convert_ordinary_to_atomic test_backward_compatibility/test_cte_distributed.py::test_cte_distributed test_backward_compatibility/test_functions.py::test_aggregate_states test_backward_compatibility/test_functions.py::test_string_functions test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column test_backward_compatibility/test_select_aggregate_alias_column.py::test_select_aggregate_alias_column test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts test_broken_part_during_merge/test.py::test_merge_and_part_corruption test_buffer_profile/test.py::test_buffer_profile test_buffer_profile/test.py::test_default_profile test_build_sets_from_multiple_threads/test.py::test_set test_cancel_freeze/test.py::test_cancel_backup test_cgroup_limit/test.py::test_cgroup_cpu_limit test_check_table/test.py::test_check_all_tables 'test_check_table/test.py::test_check_normal_table_corruption[]' 'test_check_table/test.py::test_check_replicated_table_corruption[-_0-.bin]' 'test_check_table/test.py::test_check_replicated_table_simple[-_0]' test_cleanup_after_start/test.py::test_old_dirs_cleanup test_cluster_all_replicas/test.py::test_cluster 'test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[one_shard_three_nodes]' 'test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[two_shards_three_nodes]' test_cluster_all_replicas/test.py::test_global_in 'test_cluster_all_replicas/test.py::test_skip_unavailable_replica[one_shard_three_nodes]' 'test_cluster_all_replicas/test.py::test_skip_unavailable_replica[two_shards_three_nodes]' test_cluster_discovery/test_password.py::test_connect_with_password test_compatibility_merge_tree_settings/test.py::test_check_projections_compatibility test_compatibility_merge_tree_settings/test.py::test_config_overrides_compatibility test_compression_nested_columns/test.py::test_nested_compression_codec test_concurrent_queries_for_all_users_restriction/test.py::test_exception_message test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_1 test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_50 test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_limit_reached test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_default test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_soft_limit_defined_50 test_config_substitutions/test.py::test_allow_databases test_config_substitutions/test.py::test_config test_config_substitutions/test.py::test_config_from_env_overrides test_config_substitutions/test.py::test_config_merge_from_env_overrides test_config_substitutions/test.py::test_config_multiple_zk_substitutions test_config_substitutions/test.py::test_include_config test_custom_settings/test.py::test_custom_settings -vvv -ss ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge ENV COMPOSE_HTTP_TIMEOUT 600 ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ENV DOCKER_KERBERIZED_HADOOP_TAG latest ENV DOCKER_CHANNEL stable ENV DOCKER_CLIENT_TIMEOUT 300 ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e ENV PWD /ClickHouse/tests/integration ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config ENV TZ Etc/UTC ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java ENV DOCKER_BASE_TAG 8b2301119731 ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 ENV LC_CTYPE C.UTF-8 ENV INTEGRATION_TESTS_RUN_ID 0 ENV WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 ENV PYTEST_XDIST_TESTRUNUID d21f3fb9508b4790bf0eb3faea071f00 ENV PYTEST_XDIST_WORKER gw0 ENV PYTEST_XDIST_WORKER_COUNT 10 ENV PYTEST_CURRENT_TEST test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication (setup) CLUSTER INIT base_config_dir:/clickhouse-config clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Setup Keeper Cluster name: project_name:roottestazureblobstoragezerocopyreplication-gw0. Added instance name:node1 tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env', '--project-name', 'roottestazureblobstoragezerocopyreplication-gw0', '--file', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottestazureblobstoragezerocopyreplication-gw0. Added instance name:node2 tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env', '--project-name', 'roottestazureblobstoragezerocopyreplication-gw0', '--file', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml', '--file', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Starting cluster... Running tests in /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/test.py Cluster start called. is_up=False Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Docker networks for project roottestazureblobstoragezerocopyreplication-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestazureblobstoragezerocopyreplication-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestazureblobstoragezerocopyreplication-gw0 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestazureblobstoragezerocopyreplication-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestazureblobstoragezerocopyreplication-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestazureblobstoragezerocopyreplication-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestazureblobstoragezerocopyreplication-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestazureblobstoragezerocopyreplication-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Running Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Running Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Running Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Creating Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Creating Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Creating Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Created Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Created Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Created Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Starting Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Starting Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Starting Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Started Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Started Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 Stdout:1 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node1-1/json HTTP/1.1" 200 None Volumes pruned: 1 Setup directory for instance: node1 Waiting for ClickHouse start in node1, ip: 172.16.5.7... http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node1-1/json HTTP/1.1" 200 None Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/configs/config.d/config.xml', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_gen/storage_conf.xml'] to /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/database Setup logs dir /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/configs/config.d/config.xml', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_gen/storage_conf.xml'] to /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/database Setup logs dir /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper3/coordination', 'AZURITE_PORT': '30000', 'AZURITE_STORAGE_ACCOUNT_URL': 'http://azurite1:30000/devstoreaccount1', 'AZURITE_CONNECTION_STRING': 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite1:30000/devstoreaccount1;'} stored in /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env --project-name roottestazureblobstoragezerocopyreplication-gw0 --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/docker-compose.yml pull] Skipping corrMatrix Checking quantileExactWeighted Executing query select hex(initializeAggregation('quantileExactWeightedState', 'foo')) on backward Executing query CHECK TABLE tbl on node1 Executing query SELECT sum(id) FROM default2.t on node http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Skipping quantileExactWeighted Checking sumMapFilteredWithOverflow Executing query select hex(initializeAggregation('sumMapFilteredWithOverflowState', 'foo')) on backward Executing query SELECT * FROM tbl ORDER BY id on node2 http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Executing query SELECT date, count(), sum(id) FROM default2.t GROUP BY date ORDER BY date LIMIT 2 on node Skipping sumMapFilteredWithOverflow Checking corrStable Executing query select hex(initializeAggregation('corrStableState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Executing query CHECK TABLE tbl on node2 Executing query SYSTEM FLUSH LOGS on node http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Skipping corrStable Checking covarPopMatrix Executing query select hex(initializeAggregation('covarPopMatrixState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Executing query SELECT * FROM tbl ORDER BY id on node3 http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Skipping covarPopMatrix Checking groupBitmapOr Executing query select hex(initializeAggregation('groupBitmapOrState', 'foo')) on backward Executing query SHOW TABLES FROM system on node http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Skipping groupBitmapOr Checking covarSampMatrix Executing query select hex(initializeAggregation('covarSampMatrixState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Executing query CHECK TABLE tbl on node3 Executing query SHOW TABLES FROM system on node http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Skipping covarSampMatrix Checking covarPop Executing query select hex(initializeAggregation('covarPopState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/.env --project-name roottestalternativekeeperconfig-gw4 --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/docker-compose.yml stop --timeout 20] [gw4] PASSED test_alternative_keeper_config/test.py::test_create_insert Executing query SYSTEM FLUSH LOGS on node Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/fcf137481b80c5fdc0715bb1d85ce5c0c1efff811ddd3a29e4220db9c53e8abf/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.5.5... http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d2afb9f129c509378b219fbfbae01d17c7f8b7db797b5d8c625a2e951005659b/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.5.6... http://localhost:None "GET /v1.46/containers/roottestcompatibilitymergetreesettings-gw8-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0c40bde675aba19baa00d455ac9f73b86ce7cb7cb34ff7539a3ae8982e3ce394/json HTTP/1.1" 200 None ClickHouse node3 started Executing query CREATE TABLE tp (type Int32, eventcnt UInt64, PROJECTION p (select sum(eventcnt), type group by type)) engine = ReplacingMergeTree order by type; on node1 Skipping covarPop Checking row_number Executing query select hex(initializeAggregation('row_numberState', 'foo')) on backward Executing query SELECT count() != 0 FROM system.query_log on node Executing query CREATE TABLE tp (type Int32, eventcnt UInt64, PROJECTION p (select sum(eventcnt), type group by type)) engine = ReplicatedReplacingMergeTree('/tables/tp', '0') order by type; on node1 Skipping row_number Checking kurtPop Executing query select hex(initializeAggregation('kurtPopState', 'foo')) on backward Executing query SELECT count() != 0 FROM system.part_log on node Executing query CREATE TABLE tp (type Int32, eventcnt UInt64) engine = ReplacingMergeTree order by type; on node1 Skipping kurtPop Checking kurtSamp Executing query select hex(initializeAggregation('kurtSampState', 'foo')) on backward run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -15 clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c pkill -15 clickhouse] Executing query ALTER TABLE tp ADD PROJECTION p (select sum(eventcnt), type group by type); on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Skipping kurtSamp Checking cramersV Executing query select hex(initializeAggregation('cramersVState', 'foo')) on backward Stdout:8 Skipping cramersV Checking skewSamp Executing query select hex(initializeAggregation('skewSampState', 'foo')) on backward Executing query drop table tp; on node1 Executing query CREATE TABLE tp (type Int32, eventcnt UInt64) engine = ReplicatedReplacingMergeTree('/tables/tp', '0') order by type; on node1 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '10') on node1 Skipping skewSamp Checking nonNegativeDerivative Executing query select hex(initializeAggregation('nonNegativeDerivativeState', 'foo')) on backward run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Skipping nonNegativeDerivative Checking uniqExact Executing query select hex(initializeAggregation('uniqExactState', 'foo')) on backward Executing query ALTER TABLE tp ADD PROJECTION p (select sum(eventcnt), type group by type); on node1 Executing query drop table tp; on node1 Executing query select hex(initializeAggregation('uniqExactState', 'foo')) on upstream Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_mutation Executing query CREATE TABLE tp (type Int32, eventcnt UInt64, PROJECTION p (select sum(eventcnt), type group by type)) engine = ReplacingMergeTree order by type; on node2 OK uniqExact Checking sumMapWithOverflow Executing query select hex(initializeAggregation('sumMapWithOverflowState', 'foo')) on backward run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cp /usr/bin/clickhouse /usr/share/clickhouse_original'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c cp /usr/bin/clickhouse /usr/share/clickhouse_original] Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query drop table tp; on node2 Skipping sumMapWithOverflow Checking stddevSamp Executing query select hex(initializeAggregation('stddevSampState', 'foo')) on backward Skipping stddevSamp Checking varPop Executing query select hex(initializeAggregation('varPopState', 'foo')) on backward Executing query CREATE TABLE tp (type Int32, eventcnt UInt64, PROJECTION p (select sum(eventcnt), type group by type)) engine = ReplicatedReplacingMergeTree('/tables/tp2', '0') order by type; on node2 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cp /usr/share/clickhouse_fresh /usr/bin/clickhouse && chmod 777 /usr/bin/clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c cp /usr/share/clickhouse_fresh /usr/bin/clickhouse && chmod 777 /usr/bin/clickhouse] Skipping varPop Checking quantileTimingWeighted Executing query select hex(initializeAggregation('quantileTimingWeightedState', 'foo')) on backward Executing query drop table tp; on node2 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Skipping quantileTimingWeighted Checking covarPopStable Executing query select hex(initializeAggregation('covarPopStableState', 'foo')) on backward Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Stopping Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Stopping Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Stopped Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/.env --project-name roottestaggregationmemoryefficient-gw7 --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_aggregation_memory_efficient/_instances-0-gw7/node2/docker-compose.yml down --volumes] Executing query CREATE TABLE tp (type Int32, eventcnt UInt64) engine = ReplacingMergeTree order by type; on node2 Skipping covarPopStable Checking stddevSampStable Executing query select hex(initializeAggregation('stddevSampStableState', 'foo')) on backward Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Skipping stddevSampStable Checking varSamp Executing query select hex(initializeAggregation('varSampState', 'foo')) on backward Executing query ALTER TABLE tp ADD PROJECTION p (select sum(eventcnt), type group by type); on node2 test_backup_restore_on_cluster/test.py::test_projection Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt32, y String) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}') ORDER BY y PARTITION BY x%10 on node1 Stderr: node1 Skipped - Image is already being pulled by zoo3 Stderr: zoo1 Skipped - Image is already being pulled by zoo3 Stderr: zoo2 Skipped - Image is already being pulled by zoo3 Stderr: zoo3 Pulling Stderr: zoo3 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper1/log', '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper1/config', '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper1/coordination', '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper2/log', '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper2/config', '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper2/coordination', '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper3/log', '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper3/config', '/ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/keeper3/coordination'] Command:[docker compose --project-name roottestbrokenpartduringmerge-gw6 --env-file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Skipping varSamp Checking topK Executing query select hex(initializeAggregation('topKState', 'foo')) on backward Executing query drop table tp; on node2 Executing query CREATE TABLE tp (type Int32, eventcnt UInt64) engine = ReplicatedReplacingMergeTree('/tables/tp3', '0') order by type; on node2 Executing query select hex(initializeAggregation('topKState', 'foo')) on upstream Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Stopping Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Stopping Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Stopped Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Removing Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Stopped Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Removing Stderr: Container roottestaggregationmemoryefficient-gw7-node1-1 Removed Stderr: Container roottestaggregationmemoryefficient-gw7-node2-1 Removed Stderr: Network roottestaggregationmemoryefficient-gw7_default Removing Stderr: Network roottestaggregationmemoryefficient-gw7_default Removed Cleanup called Executing query INSERT INTO tbl SELECT number, toString(number) FROM numbers(3) on node1 Docker networks for project roottestaggregationmemoryefficient-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestaggregationmemoryefficient-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestaggregationmemoryefficient-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestaggregationmemoryefficient-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestaggregationmemoryefficient-gw7 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_build_sets_from_multiple_threads/test.py::test_set Running tests in /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/test.py Cluster start called. is_up=False Docker networks for project roottestbuildsetsfrommultiplethreads-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbuildsetsfrommultiplethreads-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbuildsetsfrommultiplethreads-gw7 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbuildsetsfrommultiplethreads-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbuildsetsfrommultiplethreads-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query ALTER TABLE tp ADD PROJECTION p (select sum(eventcnt), type group by type); on node2 OK topK Checking last_value Executing query select hex(initializeAggregation('last_valueState', 'foo')) on backward Docker volumes for project roottestbuildsetsfrommultiplethreads-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbuildsetsfrommultiplethreads-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbuildsetsfrommultiplethreads-gw7 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query ALTER TABLE tbl ADD PROJECTION prjmax (SELECT MAX(x)) on node1 Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/database Setup logs dir /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/.env --project-name roottestbuildsetsfrommultiplethreads-gw7 --file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/docker-compose.yml pull] Executing query drop table tp; on node2 Executing query select hex(initializeAggregation('last_valueState', 'foo')) on upstream Executing query INSERT INTO tbl VALUES (100, 'a'), (101, 'b') on node1 [gw8] PASSED test_compatibility_merge_tree_settings/test.py::test_check_projections_compatibility test_compatibility_merge_tree_settings/test.py::test_config_overrides_compatibility Executing query CREATE TABLE tp (type Int32, eventcnt UInt64, PROJECTION p (select sum(eventcnt), type group by type)) engine = ReplacingMergeTree order by type; on node3 OK last_value Checking any Executing query select hex(initializeAggregation('anyState', 'foo')) on backward Executing query SELECT count() FROM system.projection_parts WHERE database='default' AND table='tbl' AND name='prjmax' on node1 Executing query select hex(initializeAggregation('anyState', 'foo')) on upstream Executing query CREATE TABLE tp (type Int32, eventcnt UInt64, PROJECTION p (select sum(eventcnt), type group by type)) engine = ReplicatedReplacingMergeTree('/tables/tp', '0') order by type; on node3 Stderr:time="2025-04-02T02:53:34Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestbrokenpartduringmerge-gw6_default Creating Stderr: Network roottestbrokenpartduringmerge-gw6_default Created Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Creating Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Creating Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Creating Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Created Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Created Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Created Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Starting Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Starting Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Starting Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Started Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Started Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Started Stderr:time="2025-04-02T02:53:35Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:53:35Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestbrokenpartduringmerge-gw6-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused OK any Checking anyLast Executing query select hex(initializeAggregation('anyLastState', 'foo')) on backward Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '11') on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/.env --project-name roottestcompatibilitymergetreesettings-gw8 --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/docker-compose.yml stop --timeout 20] [gw8] PASSED test_compatibility_merge_tree_settings/test.py::test_config_overrides_compatibility Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query select hex(initializeAggregation('anyLastState', 'foo')) on upstream Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused OK anyLast Checking deltaSumTimestamp Executing query select hex(initializeAggregation('deltaSumTimestampState', 'foo')) on backward Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Skipping deltaSumTimestamp Checking varSampStable Executing query select hex(initializeAggregation('varSampStableState', 'foo')) on backward Skipping varSampStable Checking nth_value Executing query select hex(initializeAggregation('nth_valueState', 'foo')) on backward Executing query SELECT count() FROM system.projection_parts WHERE database='default' AND table='tbl' AND name='prjmax' on node1 Skipping nth_value Checking argMax Executing query select hex(initializeAggregation('argMaxState', 'foo')) on backward Executing query RESTORE TABLE tbl FROM Disk('backups', '11') on node1 Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Skipping argMax Checking quantilesExact Executing query select hex(initializeAggregation('quantilesExactState', 'foo')) on backward Skipping quantilesExact Checking uniqHLL12 Executing query select hex(initializeAggregation('uniqHLL12State', 'foo')) on backward Executing query SELECT * FROM tbl ORDER BY x on node1 Executing query select hex(initializeAggregation('uniqHLL12State', 'foo')) on upstream Executing query SELECT count() FROM system.projection_parts WHERE database='default' AND table='tbl' AND name='prjmax' on node1 OK uniqHLL12 Checking quantileBFloat16 Executing query select hex(initializeAggregation('quantileBFloat16State', 'foo')) on backward [gw2] PASSED test_backup_restore_on_cluster/test.py::test_projection Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 Skipping quantileBFloat16 Checking uniq Executing query select hex(initializeAggregation('uniqState', 'foo')) on backward Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query select hex(initializeAggregation('uniqState', 'foo')) on upstream Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused OK uniq Checking min Executing query select hex(initializeAggregation('minState', 'foo')) on backward Command:[docker compose --env-file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/.env --project-name roottestclusterallreplicas-gw1 --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/docker-compose.yml stop --timeout 20] [gw1] PASSED test_cluster_all_replicas/test.py::test_skip_unavailable_replica[two_shards_three_nodes] Executing query select hex(initializeAggregation('minState', 'foo')) on upstream Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 OK min Checking sequenceNextNode Executing query select hex(initializeAggregation('sequenceNextNodeState', 'foo')) on backward Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Skipping sequenceNextNode Checking quantilesTimingWeighted Executing query select hex(initializeAggregation('quantilesTimingWeightedState', 'foo')) on backward test_backup_restore_on_cluster/test.py::test_replicated_database Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}') on node1 Skipping quantilesTimingWeighted Checking ntile Executing query select hex(initializeAggregation('ntileState', 'foo')) on backward Executing query CREATE TABLE mydb.tbl(x UInt8, y String) ENGINE=ReplicatedMergeTree ORDER BY x on node1 Skipping ntile Checking boundingRatio Executing query select hex(initializeAggregation('boundingRatioState', 'foo')) on backward Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Skipping boundingRatio Checking deltaSum Executing query select hex(initializeAggregation('deltaSumState', 'foo')) on backward Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestbrokenpartduringmerge-gw6-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SYSTEM SYNC DATABASE REPLICA mydb on node2 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestbrokenpartduringmerge-gw6-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/.env --project-name roottestbrokenpartduringmerge-gw6 --file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/.env --project-name roottestbrokenpartduringmerge-gw6 --file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate] Skipping deltaSum Checking uniqUpTo Executing query select hex(initializeAggregation('uniqUpToState', 'foo')) on backward Executing query INSERT INTO mydb.tbl VALUES (1, 'Don''t') on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "echo 'restart_with_latest_version: From version' && /usr/share/clickhouse_original server --version && echo 'To version' /usr/share/clickhouse_fresh server --version"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c echo 'restart_with_latest_version: From version' && /usr/share/clickhouse_original server --version && echo 'To version' /usr/share/clickhouse_fresh server --version] Executing query select hex(initializeAggregation('uniqUpToState', 'foo')) on upstream Executing query INSERT INTO mydb.tbl VALUES (2, 'count') on node2 Stdout:restart_with_latest_version: From version Stdout:ClickHouse server version 23.3.22.3 (official build). Stdout:To version /usr/share/clickhouse_fresh server --version run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "if [ ! -f /var/lib/clickhouse/metadata/system.sql ]; then echo 'ATTACH DATABASE system ENGINE=Ordinary' > /var/lib/clickhouse/metadata/system.sql; fi"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c if [ ! -f /var/lib/clickhouse/metadata/system.sql ]; then echo 'ATTACH DATABASE system ENGINE=Ordinary' > /var/lib/clickhouse/metadata/system.sql; fi] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "if [ ! -f /var/lib/clickhouse/metadata/default.sql ]; then echo 'ATTACH DATABASE system ENGINE=Ordinary' > /var/lib/clickhouse/metadata/default.sql; fi"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c if [ ! -f /var/lib/clickhouse/metadata/default.sql ]; then echo 'ATTACH DATABASE system ENGINE=Ordinary' > /var/lib/clickhouse/metadata/default.sql; fi] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- '] Command:[docker exec -u 0 roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] OK uniqUpTo Checking windowFunnel Executing query select hex(initializeAggregation('windowFunnelState', 'foo')) on backward Stdout:387 Executing query select 20 on node Executing query INSERT INTO mydb.tbl VALUES (3, 'your') on node1 Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/.env --project-name roottestcompatibilitymergetreesettings-gw8 --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_compatibility_merge_tree_settings/_instances-0-gw8/node3/docker-compose.yml down --volumes] Skipping windowFunnel Checking retention Executing query select hex(initializeAggregation('retentionState', 'foo')) on backward Executing query INSERT INTO mydb.tbl VALUES (4, 'chickens') on node2 Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Running Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Running Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Running Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Creating Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Created Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Starting Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbrokenpartduringmerge-gw6-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbrokenpartduringmerge-gw6-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestbrokenpartduringmerge-gw6-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Skipping retention Checking sequenceMatch Executing query select hex(initializeAggregation('sequenceMatchState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Executing query OPTIMIZE TABLE mydb.tbl ON CLUSTER 'cluster' FINAL on node1 http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Skipping sequenceMatch Checking uniqTheta Executing query select hex(initializeAggregation('uniqThetaState', 'foo')) on backward Executing query select 20 on node http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Removing Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Removing Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Removing Stderr: Container roottestcompatibilitymergetreesettings-gw8-node2-1 Removed Stderr: Container roottestcompatibilitymergetreesettings-gw8-node1-1 Removed Stderr: Container roottestcompatibilitymergetreesettings-gw8-node3-1 Removed Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Stopping Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Removing Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Removing Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Stopped Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Removing Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo3-1 Removed Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo1-1 Removed Stderr: Container roottestcompatibilitymergetreesettings-gw8-zoo2-1 Removed Stderr: Network roottestcompatibilitymergetreesettings-gw8_default Removing Stderr: Network roottestcompatibilitymergetreesettings-gw8_default Removed Cleanup called http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Docker networks for project roottestcompatibilitymergetreesettings-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcompatibilitymergetreesettings-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcompatibilitymergetreesettings-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcompatibilitymergetreesettings-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcompatibilitymergetreesettings-gw8 Trying to prune unused networks... Executing query select hex(initializeAggregation('uniqThetaState', 'foo')) on upstream Trying to prune unused images... Command:[docker image prune -f] http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_cgroup_limit/test.py::test_cgroup_cpu_limit Command: docker run --rm --cpus 1 --volume /clickhouse:/usr/bin/clickhouse ubuntu:22.04 sh -c clickhouse local -q "select value from system.settings where name='max_threads'" Unable to find image 'ubuntu:22.04' locally http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl on node1 OK uniqTheta Checking quantilesExactWeighted Executing query select hex(initializeAggregation('quantilesExactWeightedState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Skipping quantilesExactWeighted Checking max Executing query select hex(initializeAggregation('maxState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Executing query SELECT * FROM mydb.tbl ORDER BY x on node1 Executing query select 20 on node http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Executing query select hex(initializeAggregation('maxState', 'foo')) on upstream http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None Executing query SELECT * FROM mydb.tbl ORDER BY x on node2 Executing query SHOW CREATE DATABASE default2 on node http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7dfdf01e65a0b7d0ef8ed933640da5eaae1bee26db8b4cdbb78b3109afbd771b/json HTTP/1.1" 200 None ClickHouse node1 started Executing query CREATE TABLE replicated_mt(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/replicated_mt', 'node1') ORDER BY id SETTINGS cleanup_delay_period=1, cleanup_delay_period_random_add=1, cleanup_thread_preferred_points_per_iteration=0; on node1 OK max Checking quantilesBFloat16Weighted Executing query select hex(initializeAggregation('quantilesBFloat16WeightedState', 'foo')) on backward Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster' TO Disk('backups', '12') SETTINGS replica_num=2 on node1 Executing query SHOW CREATE DATABASE system on node Skipping quantilesBFloat16Weighted Checking quantileBFloat16Weighted Executing query select hex(initializeAggregation('quantileBFloat16WeightedState', 'foo')) on backward Executing query SYSTEM STOP REPLICATION QUEUES replicated_mt on node1 Executing query SHOW TABLES FROM system on node Skipping quantileBFloat16Weighted Checking quantilesTDigestWeighted Executing query select hex(initializeAggregation('quantilesTDigestWeightedState', 'foo')) on backward Executing query DROP DATABASE mydb ON CLUSTER 'cluster' SYNC on node1 Executing query INSERT INTO replicated_mt SELECT toDate('2019-10-01'), number, number * number FROM numbers (0, 100000) on node1 Executing query SHOW TABLES FROM system on node Skipping quantilesTDigestWeighted Checking groupBitAnd Executing query select hex(initializeAggregation('groupBitAndState', 'foo')) on backward Executing query SYSTEM FLUSH LOGS on node Skipping groupBitAnd Checking quantileTDigest Executing query select hex(initializeAggregation('quantileTDigestState', 'foo')) on backward Executing query INSERT INTO replicated_mt SELECT toDate('2019-10-01'), number, number * number FROM numbers (100000, 100000) on node1 Executing query RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM Disk('backups', '12') on node1 Skipping quantileTDigest Checking quantileTDigestWeighted Executing query select hex(initializeAggregation('quantileTDigestWeightedState', 'foo')) on backward Executing query INSERT INTO replicated_mt SELECT toDate('2019-10-01'), number, number * number FROM numbers (200000, 100000) on node1 Skipping quantileTDigestWeighted Checking quantileDeterministic Executing query select hex(initializeAggregation('quantileDeterministicState', 'foo')) on backward Executing query INSERT INTO replicated_mt SELECT toDate('2019-10-01'), number, number * number FROM numbers (300000, 100000) on node1 Skipping quantileDeterministic Checking quantilesTDigest Executing query select hex(initializeAggregation('quantilesTDigestState', 'foo')) on backward Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl on node1 Executing query SELECT COUNT() FROM system.parts WHERE table='replicated_mt' AND active=1 on node1 Executing query SHOW TABLES FROM system on node Skipping quantilesTDigest Checking stochasticLogisticRegression Executing query select hex(initializeAggregation('stochasticLogisticRegressionState', 'foo')) on backward Skipping stochasticLogisticRegression Checking argMin Executing query select hex(initializeAggregation('argMinState', 'foo')) on backward Executing query SELECT path FROM system.parts WHERE table = 'replicated_mt' and name = 'all_3_3_0' on node1 Executing query SHOW TABLES FROM system on node Executing query SELECT * FROM mydb.tbl ORDER BY x on node1 Skipping argMin Checking avg Executing query select hex(initializeAggregation('avgState', 'foo')) on backward run container_id:roottestbrokenpartduringmerge-gw6-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'cd /var/lib/clickhouse/store/78f/78fb9d47-3243-49c3-95e6-60b1747793ea/all_3_3_0/ && ls *.bin | head -n 1'] Command:[docker exec roottestbrokenpartduringmerge-gw6-node1-1 bash -c cd /var/lib/clickhouse/store/78f/78fb9d47-3243-49c3-95e6-60b1747793ea/all_3_3_0/ && ls *.bin | head -n 1] Executing query SELECT count() != 0 FROM system.query_log_0 on node Executing query SELECT * FROM mydb.tbl ORDER BY x on node2 Stdout:data.bin run container_id:roottestbrokenpartduringmerge-gw6-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "cd /var/lib/clickhouse/store/78f/78fb9d47-3243-49c3-95e6-60b1747793ea/all_3_3_0/ && ls *.bin | head -n 1 | xargs -I{} sh -c 'truncate -s -1 $1' -- {}"] Command:[docker exec --privileged roottestbrokenpartduringmerge-gw6-node1-1 bash -c cd /var/lib/clickhouse/store/78f/78fb9d47-3243-49c3-95e6-60b1747793ea/all_3_3_0/ && ls *.bin | head -n 1 | xargs -I{} sh -c 'truncate -s -1 $1' -- {}] Executing query CHECK TABLE replicated_mt on node1 Executing query OPTIMIZE TABLE replicated_mt FINAL on node1 Skipping avg Checking covarSampStable Executing query select hex(initializeAggregation('covarSampStableState', 'foo')) on backward Executing query SELECT count() != 0 FROM system.part_log_0 on node Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database Executing query SYSTEM START REPLICATION QUEUES replicated_mt on node1 Skipping covarSampStable Checking quantilesTiming Executing query select hex(initializeAggregation('quantilesTimingState', 'foo')) on backward Executing query SELECT date, count(), sum(id) FROM default2.t GROUP BY date ORDER BY date LIMIT 2 on node Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Skipping quantilesTiming Checking welchTTest Executing query select hex(initializeAggregation('welchTTestState', 'foo')) on backward Executing query SELECT name FROM system.databases ORDER BY name on node Skipping welchTTest Checking covarSamp Executing query select hex(initializeAggregation('covarSampState', 'foo')) on backward Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query ALTER TABLE replicated_mt UPDATE value = 7 WHERE 1 on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'grep -a "" /var/log/clickhouse-server/clickhouse-server.log | wc -l'] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c grep -a "" /var/log/clickhouse-server/clickhouse-server.log | wc -l] Skipping covarSamp Checking varPopStable Executing query select hex(initializeAggregation('varPopStableState', 'foo')) on backward Stdout:0 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'grep -a " Database" /var/log/clickhouse-server/clickhouse-server.log | wc -l'] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c grep -a " Database" /var/log/clickhouse-server/clickhouse-server.log | wc -l] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/.env --project-name roottestbuildsetsfrommultiplethreads-gw7 --file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/.env --project-name roottestbuildsetsfrommultiplethreads-gw7 --file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/docker-compose.yml up -d --no-recreate] Stdout:0 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'grep -a "always include the lines below" /var/log/clickhouse-server/clickhouse-server.log | wc -l'] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c grep -a "always include the lines below" /var/log/clickhouse-server/clickhouse-server.log | wc -l] Stdout:0 Executing query CREATE DATABASE ordinary ENGINE=Ordinary on node Skipping varPopStable Checking quantileTiming Executing query select hex(initializeAggregation('quantileTimingState', 'foo')) on backward 22.04: Pulling from library/ubuntu Executing query SELECT sum(value) FROM replicated_mt on node1 9cb31e2e37ea: Already exists Digest: sha256:ed1544e454989078f5dec1bfdabd8c5cc9c48e0705d07b678ab6ae3fb61952d2 Status: Downloaded newer image for ubuntu:22.04 Executing query CREATE DATABASE other ENGINE=Ordinary on node Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Skipping quantileTiming Checking leadInFrame Executing query select hex(initializeAggregation('leadInFrameState', 'foo')) on backward Executing query DROP TABLE replicated_mt SYNC on node1 Executing query CREATE DATABASE `.o r d i n a r y.` ENGINE=Ordinary on node Skipping leadInFrame Checking quantilesExactExclusive Executing query select hex(initializeAggregation('quantilesExactExclusiveState', 'foo')) on backward Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Stderr: Network roottestbuildsetsfrommultiplethreads-gw7_default Creating Stderr: Network roottestbuildsetsfrommultiplethreads-gw7_default Created Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Creating Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Created Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Starting Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestbuildsetsfrommultiplethreads-gw7-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestbuildsetsfrommultiplethreads-gw7-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.2.2... http://localhost:None "GET /v1.46/containers/roottestbuildsetsfrommultiplethreads-gw7-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None Executing query CREATE DATABASE atomic ENGINE=Atomic on node Skipping quantilesExactExclusive Checking maxIntersectionsPosition Executing query select hex(initializeAggregation('maxIntersectionsPositionState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/.env --project-name roottestbrokenpartduringmerge-gw6 --file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] [gw6] PASSED test_broken_part_during_merge/test.py::test_merge_and_part_corruption http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None test_backup_restore_on_cluster/test.py::test_replicated_database_async Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}') on node1 Skipping maxIntersectionsPosition Checking quantilesBFloat16 Executing query select hex(initializeAggregation('quantilesBFloat16State', 'foo')) on backward Executing query CREATE DATABASE mem ENGINE=Memory on node http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None Command: docker run --rm --cpus 2 --volume /clickhouse:/usr/bin/clickhouse ubuntu:22.04 sh -c clickhouse local -q "select value from system.settings where name='max_threads'" Skipping quantilesBFloat16 Checking quantilesExactInclusive Executing query select hex(initializeAggregation('quantilesExactInclusiveState', 'foo')) on backward Executing query CREATE DATABASE lazy ENGINE=Lazy(1) on node http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None Executing query CREATE TABLE mydb.tbl(x UInt8) ENGINE=ReplicatedMergeTree ORDER BY x on node1 http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None Executing query CREATE TABLE ordinary.t1 (n int) ENGINE=Memory on node Skipping quantilesExactInclusive Checking quantileExactInclusive Executing query select hex(initializeAggregation('quantileExactInclusiveState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None Executing query CREATE TABLE mydb.tbl2(y String) ENGINE=ReplicatedMergeTree ORDER BY y on node1 Executing query CREATE TABLE ordinary.mt1 (n int) ENGINE=MergeTree order by n on node Skipping quantileExactInclusive Checking quantileInterpolatedWeighted Executing query select hex(initializeAggregation('quantileInterpolatedWeightedState', 'foo')) on backward http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Stopping Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Stopping Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Stopping Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Stopped Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Stopped Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command: docker run --rm --cpus 4 --volume /clickhouse:/usr/bin/clickhouse ubuntu:22.04 sh -c clickhouse local -q "select value from system.settings where name='max_threads'" Command:[bash -c [ -f /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Skipping quantileInterpolatedWeighted Checking quantileExactHigh Executing query select hex(initializeAggregation('quantileExactHighState', 'foo')) on backward Command:[bash -c [ -f /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/.env --project-name roottestalternativekeeperconfig-gw4 --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_alternative_keeper_config/_instances-0-gw4/node3/docker-compose.yml down --volumes] Executing query CREATE TABLE ordinary.mt2 (n int) ENGINE=MergeTree order by n on node http://localhost:None "GET /v1.46/containers/296faf499703091d4439f46ad0b069923ee55947895e17a5c8a14ed47e4a6cb7/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE TABLE 02581_trips (id UInt32, description String, id2 UInt32) ENGINE = MergeTree PRIMARY KEY id ORDER BY id; INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips SELECT number, '', number FROM numbers(1); INSERT INTO 02581_trips on node Executing query SYSTEM SYNC DATABASE REPLICA mydb on node2 Skipping quantileExactHigh Checking groupBitmap Executing query select hex(initializeAggregation('groupBitmapState', 'foo')) on backward Executing query INSERT INTO mydb.tbl VALUES (1) on node1 Executing query CREATE TABLE ordinary.rmt1 (n int, m int) ENGINE=ReplicatedMergeTree('/test/rmt1/ordinary', '1') order by n on node Skipping groupBitmap Checking quantilesInterpolatedWeighted Executing query select hex(initializeAggregation('quantilesInterpolatedWeightedState', 'foo')) on backward Executing query INSERT INTO mydb.tbl VALUES (22) on node1 Executing query CREATE TABLE ordinary.rmt2 (n int, m int) ENGINE=ReplicatedMergeTree('/test/ordinary/rmt2', '1') order by n on node Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Stopping Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Stopping Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Stopping Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Stopped Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Removing Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Stopped Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Removing Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Stopped Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Removing Stderr: Container roottestalternativekeeperconfig-gw4-node1-1 Removed Stderr: Container roottestalternativekeeperconfig-gw4-node2-1 Removed Stderr: Container roottestalternativekeeperconfig-gw4-node3-1 Removed Stderr: Network roottestalternativekeeperconfig-gw4_default Removing Stderr: Network roottestalternativekeeperconfig-gw4_default Removed Cleanup called Docker networks for project roottestalternativekeeperconfig-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestalternativekeeperconfig-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestalternativekeeperconfig-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestalternativekeeperconfig-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestalternativekeeperconfig-gw4 Trying to prune unused networks... Skipping quantilesInterpolatedWeighted Checking quantile Executing query select hex(initializeAggregation('quantileState', 'foo')) on backward Trying to prune unused images... Command:[docker image prune -f] Executing query ALTER TABLE `02581_trips` UPDATE description = 'a' WHERE id IN (SELECT CAST(number * 10, 'UInt32') FROM numbers(10e9)) SETTINGS mutations_sync = 2 on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup Running tests in /ClickHouse/tests/integration/test_backup_restore_new/test_shutdown_wait_backup.py Cluster start called. is_up=False Docker networks for project roottestbackuprestorenewshutdownwaitbackup-gw4 are NETWORK ID NAME DRIVER SCOPE Executing query INSERT INTO mydb.tbl2 VALUES ('a') on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "sed --follow-symlinks -i 's|/test/ordinary/rmt2|/test/{database}/{table}|' /var/lib/clickhouse/metadata/ordinary/rmt2.sql"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c sed --follow-symlinks -i 's|/test/ordinary/rmt2|/test/{database}/{table}|' /var/lib/clickhouse/metadata/ordinary/rmt2.sql] Docker containers for project roottestbackuprestorenewshutdownwaitbackup-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackuprestorenewshutdownwaitbackup-gw4 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbackuprestorenewshutdownwaitbackup-gw4 are NETWORK ID NAME DRIVER SCOPE Executing query CREATE MATERIALIZED VIEW ordinary.mv1 (n int) ENGINE=ReplicatedMergeTree('/test/ordinary/mv1/', '1') order by n AS SELECT n FROM ordinary.rmt1 on node Docker containers for project roottestbackuprestorenewshutdownwaitbackup-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackuprestorenewshutdownwaitbackup-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackuprestorenewshutdownwaitbackup-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackuprestorenewshutdownwaitbackup-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Command: docker run --rm --cpus 2.8 --volume /clickhouse:/usr/bin/clickhouse ubuntu:22.04 sh -c clickhouse local -q "select value from system.settings where name='max_threads'" Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_backup_restore_new/configs/backups_disk.xml', '/ClickHouse/tests/integration/test_backup_restore_new/configs/slow_backups.xml'] to /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/database Setup logs dir /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" external_dir_abs_path=/ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/backups Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/.env --project-name roottestbackuprestorenewshutdownwaitbackup-gw4 --file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/docker-compose.yml pull] Skipping quantile Checking groupArraySample Executing query select hex(initializeAggregation('groupArraySampleState', 'foo')) on backward Executing query INSERT INTO mydb.tbl2 VALUES ('bb') on node2 Executing query CREATE MATERIALIZED VIEW ordinary.mv2 (n int) ENGINE=MergeTree order by n AS SELECT n FROM ordinary.rmt2 on node Executing query CREATE DICTIONARY ordinary.d1 (n int DEFAULT 0, m int DEFAULT 1) PRIMARY KEY n SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'rmt1' PASSWORD '' DB 'ordinary')) LIFETIME(MIN 1 MAX 10) LAYOUT(FLAT()) on node Skipping groupArraySample Checking quantilesExactLow Executing query select hex(initializeAggregation('quantilesExactLowState', 'foo')) on backward Executing query OPTIMIZE TABLE mydb.tbl ON CLUSTER 'cluster' FINAL on node1 Executing query CREATE DICTIONARY ordinary.d2 (n int DEFAULT 0, m int DEFAULT 1) PRIMARY KEY n SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'rmt2' PASSWORD '' DB 'ordinary')) LIFETIME(MIN 1 MAX 10) LAYOUT(FLAT()) on node Skipping quantilesExactLow Checking groupUniqArray Executing query select hex(initializeAggregation('groupUniqArrayState', 'foo')) on backward Executing query OPTIMIZE TABLE mydb.tbl2 ON CLUSTER 'cluster' FINAL on node1 [gw8] PASSED test_cgroup_limit/test.py::test_cgroup_cpu_limit Executing query CREATE TABLE ordinary.merge (n int) ENGINE=Merge('ordinary', '(mt)|(mv)') on node Executing query select hex(initializeAggregation('groupUniqArrayState', 'foo')) on upstream Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl on node1 Executing query CREATE TABLE ordinary.detached (n int) ENGINE=Log on node OK groupUniqArray Checking sumWithOverflow Executing query select hex(initializeAggregation('sumWithOverflowState', 'foo')) on backward Executing query INSERT INTO ordinary.mt1 (n) VALUES (8) on node Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl2 on node1 Skipping sumWithOverflow Checking sequenceCount Executing query select hex(initializeAggregation('sequenceCountState', 'foo')) on backward Skipping sequenceCount Checking quantilesDeterministic Executing query select hex(initializeAggregation('quantilesDeterministicState', 'foo')) on backward Executing query INSERT INTO ordinary.mt2 (n) VALUES (8) on node Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Stopping Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Stopped Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Stopping Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Stopping Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Stopping Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Stopped Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Stopped Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/.env --project-name roottestbrokenpartduringmerge-gw6 --file /ClickHouse/tests/integration/test_broken_part_during_merge/_instances-0-gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster' TO Disk('backups', '13') ASYNC on node1 Skipping quantilesDeterministic Checking quantilesExactHigh Executing query select hex(initializeAggregation('quantilesExactHighState', 'foo')) on backward Executing query INSERT INTO ordinary.rmt1 (n) VALUES (8) on node Executing query SELECT status, error FROM system.backups WHERE id='8c27c159-43e3-4579-917c-abc21ccf1466' on node1 Skipping quantilesExactHigh Checking sumCount Executing query select hex(initializeAggregation('sumCountState', 'foo')) on backward Executing query INSERT INTO ordinary.rmt2 (n) VALUES (8) on node Executing query DROP DATABASE mydb ON CLUSTER 'cluster' SYNC on node1 Skipping sumCount Checking groupArrayInsertAt Executing query select hex(initializeAggregation('groupArrayInsertAtState', 'foo')) on backward Executing query INSERT INTO ordinary.mv1 (n) VALUES (8) on node Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Stopping Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Stopped Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Removing Stderr: Container roottestbrokenpartduringmerge-gw6-node1-1 Removed Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Stopping Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Stopping Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Stopping Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Stopped Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Removing Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Stopped Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Stopped Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Removing Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Removing Stderr: Container roottestbrokenpartduringmerge-gw6-zoo1-1 Removed Stderr: Container roottestbrokenpartduringmerge-gw6-zoo3-1 Removed Stderr: Container roottestbrokenpartduringmerge-gw6-zoo2-1 Removed Stderr: Network roottestbrokenpartduringmerge-gw6_default Removing Stderr: Network roottestbrokenpartduringmerge-gw6_default Removed Cleanup called Docker networks for project roottestbrokenpartduringmerge-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbrokenpartduringmerge-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Skipping groupArrayInsertAt Checking quantileExact Executing query select hex(initializeAggregation('quantileExactState', 'foo')) on backward Docker volumes for project roottestbrokenpartduringmerge-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbrokenpartduringmerge-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbrokenpartduringmerge-gw6 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_concurrent_queries_for_all_users_restriction/test.py::test_exception_message Running tests in /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/test.py Cluster start called. is_up=False Docker networks for project roottestconcurrentqueriesforallusersrestriction-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestconcurrentqueriesforallusersrestriction-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query INSERT INTO ordinary.mv2 (n) VALUES (8) on node Docker volumes for project roottestconcurrentqueriesforallusersrestriction-gw6 are DRIVER VOLUME NAME Cleanup called Executing query RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM Disk('backups', '13') ASYNC on node1 Docker networks for project roottestconcurrentqueriesforallusersrestriction-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestconcurrentqueriesforallusersrestriction-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestconcurrentqueriesforallusersrestriction-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestconcurrentqueriesforallusersrestriction-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestconcurrentqueriesforallusersrestriction-gw6 Trying to prune unused networks... Skipping quantileExact Checking anyHeavy Executing query select hex(initializeAggregation('anyHeavyState', 'foo')) on backward Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/database Setup logs dir /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/.env --project-name roottestconcurrentqueriesforallusersrestriction-gw6 --file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/docker-compose.yml pull] Executing query INSERT INTO ordinary.detached (n) VALUES (8) on node Executing query SELECT status, error FROM system.backups WHERE id='1c2a668d-c67c-422b-aac4-f674fe3d1a75' on node1 Executing query select hex(initializeAggregation('anyHeavyState', 'foo')) on upstream Executing query CREATE TABLE other.t1 (n int) ENGINE=Memory on node Executing query select finalizeAggregation(unhex('04000000666F6F000100000000000000')::AggregateFunction(anyHeavy, String)) on backward Executing query CREATE TABLE other.mt1 (n int) ENGINE=MergeTree order by n on node Executing query select finalizeAggregation(unhex('04000000666F6F000000000000000000')::AggregateFunction(anyHeavy, String)) on upstream Executing query CREATE TABLE other.mt2 (n int) ENGINE=MergeTree order by n on node Executing query CREATE TABLE other.rmt1 (n int, m int) ENGINE=ReplicatedMergeTree('/test/rmt1/other', '1') order by n on node Executing query SELECT status, error FROM system.backups WHERE id='1c2a668d-c67c-422b-aac4-f674fe3d1a75' on node1 OK anyHeavy (but different intermediate states) Checking theilsU Executing query select hex(initializeAggregation('theilsUState', 'foo')) on backward Stderr: Container roottestclusterallreplicas-gw1-node2-1 Stopping Stderr: Container roottestclusterallreplicas-gw1-node1-1 Stopping Stderr: Container roottestclusterallreplicas-gw1-node2-1 Stopped Stderr: Container roottestclusterallreplicas-gw1-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/.env --project-name roottestclusterallreplicas-gw1 --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_cluster_all_replicas/_instances-0-gw1/node2/docker-compose.yml down --volumes] Executing query CREATE TABLE other.rmt2 (n int, m int) ENGINE=ReplicatedMergeTree('/test/other/rmt2', '1') order by n on node Command:[docker compose --env-file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/.env --project-name roottestbuildsetsfrommultiplethreads-gw7 --file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/docker-compose.yml stop --timeout 20] [gw7] PASSED test_build_sets_from_multiple_threads/test.py::test_set Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl on node1 Skipping theilsU Checking maxMappedArrays Executing query select hex(initializeAggregation('maxMappedArraysState', 'foo')) on backward Skipping maxMappedArrays Checking stddevPop Executing query select hex(initializeAggregation('stddevPopState', 'foo')) on backward run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "sed --follow-symlinks -i 's|/test/other/rmt2|/test/{database}/{table}|' /var/lib/clickhouse/metadata/other/rmt2.sql"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c sed --follow-symlinks -i 's|/test/other/rmt2|/test/{database}/{table}|' /var/lib/clickhouse/metadata/other/rmt2.sql] Executing query CREATE MATERIALIZED VIEW other.mv1 (n int) ENGINE=ReplicatedMergeTree('/test/other/mv1/', '1') order by n AS SELECT n FROM other.rmt1 on node Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl2 on node1 Skipping stddevPop Checking groupArrayLast Executing query select hex(initializeAggregation('groupArrayLastState', 'foo')) on backward Stderr: Container roottestclusterallreplicas-gw1-node1-1 Stopping Stderr: Container roottestclusterallreplicas-gw1-node2-1 Stopping Stderr: Container roottestclusterallreplicas-gw1-node1-1 Stopped Stderr: Container roottestclusterallreplicas-gw1-node1-1 Removing Stderr: Container roottestclusterallreplicas-gw1-node2-1 Stopped Stderr: Container roottestclusterallreplicas-gw1-node2-1 Removing Stderr: Container roottestclusterallreplicas-gw1-node2-1 Removed Stderr: Container roottestclusterallreplicas-gw1-node1-1 Removed Stderr: Network roottestclusterallreplicas-gw1_default Removing Stderr: Network roottestclusterallreplicas-gw1_default Removed Cleanup called Docker networks for project roottestclusterallreplicas-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestclusterallreplicas-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestclusterallreplicas-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestclusterallreplicas-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query CREATE MATERIALIZED VIEW other.mv2 (n int) ENGINE=MergeTree order by n AS SELECT n FROM other.rmt2 on node Unstopped containers: {} No running containers for project: roottestclusterallreplicas-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Skipping groupArrayLast Checking groupArray Executing query select hex(initializeAggregation('groupArrayState', 'foo')) on backward Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query SELECT * FROM mydb.tbl ORDER BY x on node1 Stdout:1 Volumes pruned: 1 test_backward_compatibility/test_select_aggregate_alias_column.py::test_select_aggregate_alias_column Running tests in /ClickHouse/tests/integration/test_backward_compatibility/test_select_aggregate_alias_column.py Cluster start called. is_up=False Docker networks for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query CREATE DICTIONARY other.d1 (n int DEFAULT 0, m int DEFAULT 1) PRIMARY KEY n SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'rmt1' PASSWORD '' DB 'other')) LIFETIME(MIN 1 MAX 10) LAYOUT(FLAT()) on node Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query select hex(initializeAggregation('groupArrayState', 'foo')) on upstream Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/.env --project-name roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/docker-compose.yml pull] Executing query SELECT * FROM mydb.tbl2 ORDER BY y on node2 Executing query CREATE DICTIONARY other.d2 (n int DEFAULT 0, m int DEFAULT 1) PRIMARY KEY n SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'rmt2' PASSWORD '' DB 'other')) LIFETIME(MIN 1 MAX 10) LAYOUT(FLAT()) on node OK groupArray Checking first_value Executing query select hex(initializeAggregation('first_valueState', 'foo')) on backward Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_async Executing query CREATE TABLE other.merge (n int) ENGINE=Merge('other', '(mt)|(mv)') on node Executing query select hex(initializeAggregation('first_valueState', 'foo')) on upstream Executing query CREATE TABLE other.detached (n int) ENGINE=Log on node Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 OK first_value Checking exponentialTimeDecayedAvg Executing query select hex(initializeAggregation('exponentialTimeDecayedAvgState', 'foo')) on backward Executing query INSERT INTO other.mt1 (n) VALUES (5) on node Skipping exponentialTimeDecayedAvg Checking quantileExactExclusive Executing query select hex(initializeAggregation('quantileExactExclusiveState', 'foo')) on backward Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query INSERT INTO other.mt2 (n) VALUES (5) on node Skipping quantileExactExclusive Checking count Executing query select hex(initializeAggregation('countState', 'foo')) on backward Executing query INSERT INTO other.rmt1 (n) VALUES (5) on node Executing query select hex(initializeAggregation('countState', 'foo')) on upstream Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query INSERT INTO other.rmt2 (n) VALUES (5) on node OK count Checking stddevPopStable Executing query select hex(initializeAggregation('stddevPopStableState', 'foo')) on backward Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query INSERT INTO other.mv1 (n) VALUES (5) on node Skipping stddevPopStable Checking quantileExactLow Executing query select hex(initializeAggregation('quantileExactLowState', 'foo')) on backward Executing query INSERT INTO other.mv2 (n) VALUES (5) on node Skipping quantileExactLow Checking avgWeighted Executing query select hex(initializeAggregation('avgWeightedState', 'foo')) on backward test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}') on node1 Skipping avgWeighted Checking analysisOfVariance Executing query select hex(initializeAggregation('analysisOfVarianceState', 'foo')) on backward Executing query INSERT INTO other.detached (n) VALUES (5) on node Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Skipping analysisOfVariance Checking groupBitAnd Executing query select hex(initializeAggregation('groupBitAndState', 'foo')) on backward Executing query CREATE TABLE atomic.t1 (n int) ENGINE=Memory on node Executing query CREATE TABLE mydb.tbl(x UInt8, y String) ENGINE=ReplicatedMergeTree ORDER BY x on node1 Skipping groupBitAnd Checking varSamp Executing query select hex(initializeAggregation('varSampState', 'foo')) on backward Executing query CREATE TABLE atomic.mt1 (n int) ENGINE=MergeTree order by n on node Skipping varSamp Checking quantileTimingWeighted Executing query select hex(initializeAggregation('quantileTimingWeightedState', 'foo')) on backward Executing query SYSTEM SYNC DATABASE REPLICA mydb on node2 Executing query CREATE TABLE atomic.mt2 (n int) ENGINE=MergeTree order by n on node Skipping quantileTimingWeighted Checking covarSamp Executing query select hex(initializeAggregation('covarSampState', 'foo')) on backward Executing query SYSTEM STOP MERGES mydb.tbl on node1 Executing query CREATE TABLE atomic.rmt1 (n int, m int) ENGINE=ReplicatedMergeTree('/test/rmt1/atomic', '1') order by n on node Skipping covarSamp Checking varPop Executing query select hex(initializeAggregation('varPopState', 'foo')) on backward Executing query SYSTEM STOP MERGES mydb.tbl on node2 Skipping varPop Checking quantileBFloat16 Executing query select hex(initializeAggregation('quantileBFloat16State', 'foo')) on backward Executing query SYSTEM STOP FETCHES mydb.tbl on node1 Executing query CREATE TABLE atomic.rmt2 (n int, m int) ENGINE=ReplicatedMergeTree('/test/atomic/rmt2', '1') order by n on node Skipping quantileBFloat16 Checking quantileTDigest Executing query select hex(initializeAggregation('quantileTDigestState', 'foo')) on backward Executing query SYSTEM STOP FETCHES mydb.tbl on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "sed --follow-symlinks -i 's|/test/atomic/rmt2|/test/{database}/{table}|' /var/lib/clickhouse/metadata/atomic/rmt2.sql"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c sed --follow-symlinks -i 's|/test/atomic/rmt2|/test/{database}/{table}|' /var/lib/clickhouse/metadata/atomic/rmt2.sql] Executing query CREATE MATERIALIZED VIEW atomic.mv1 (n int) ENGINE=ReplicatedMergeTree('/test/atomic/mv1/', '1') order by n AS SELECT n FROM atomic.rmt1 on node Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Stopping Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/.env --project-name roottestbuildsetsfrommultiplethreads-gw7 --file /ClickHouse/tests/integration/test_build_sets_from_multiple_threads/_instances-0-gw7/node/docker-compose.yml down --volumes] Skipping quantileTDigest Checking quantileTiming Executing query select hex(initializeAggregation('quantileTimingState', 'foo')) on backward Executing query INSERT INTO mydb.tbl VALUES (1, 'a') on node1 Executing query CREATE MATERIALIZED VIEW atomic.mv2 (n int) ENGINE=MergeTree order by n AS SELECT n FROM atomic.rmt2 on node Skipping quantileTiming Checking quantileExactHigh Executing query select hex(initializeAggregation('quantileExactHighState', 'foo')) on backward Executing query INSERT INTO mydb.tbl VALUES (2, 'b') on node1 Executing query CREATE DICTIONARY atomic.d1 (n int DEFAULT 0, m int DEFAULT 1) PRIMARY KEY n SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'rmt1' PASSWORD '' DB 'atomic')) LIFETIME(MIN 1 MAX 10) LAYOUT(FLAT()) on node Skipping quantileExactHigh Checking quantileInterpolatedWeighted Executing query select hex(initializeAggregation('quantileInterpolatedWeightedState', 'foo')) on backward Executing query INSERT INTO mydb.tbl VALUES (3, 'x') on node2 Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Stopping Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Stopped Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Removing Stderr: Container roottestbuildsetsfrommultiplethreads-gw7-node-1 Removed Stderr: Network roottestbuildsetsfrommultiplethreads-gw7_default Removing Stderr: Network roottestbuildsetsfrommultiplethreads-gw7_default Removed Cleanup called Docker networks for project roottestbuildsetsfrommultiplethreads-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbuildsetsfrommultiplethreads-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query CREATE DICTIONARY atomic.d2 (n int DEFAULT 0, m int DEFAULT 1) PRIMARY KEY n SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'default' TABLE 'rmt2' PASSWORD '' DB 'atomic')) LIFETIME(MIN 1 MAX 10) LAYOUT(FLAT()) on node Docker volumes for project roottestbuildsetsfrommultiplethreads-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbuildsetsfrommultiplethreads-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbuildsetsfrommultiplethreads-gw7 Trying to prune unused networks... Skipping quantileInterpolatedWeighted Checking groupBitXor Executing query select hex(initializeAggregation('groupBitXorState', 'foo')) on backward Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_custom_settings/test.py::test_custom_settings Running tests in /ClickHouse/tests/integration/test_custom_settings/test.py Cluster start called. is_up=False Executing query INSERT INTO mydb.tbl VALUES (4, 'y') on node2 Docker networks for project roottestcustomsettings-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcustomsettings-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcustomsettings-gw7 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestcustomsettings-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcustomsettings-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcustomsettings-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcustomsettings-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcustomsettings-gw7 Trying to prune unused networks... Executing query CREATE TABLE atomic.merge (n int) ENGINE=Merge('atomic', '(mt)|(mv)') on node Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Skipping groupBitXor Checking quantile Executing query select hex(initializeAggregation('quantileState', 'foo')) on backward Stdout:1 Volumes pruned: 1 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/database Setup logs dir /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/.env --project-name roottestcustomsettings-gw7 --file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/docker-compose.yml pull] Executing query SELECT * FROM mydb.tbl ORDER BY x on node2 Skipping quantile Checking covarPop Executing query select hex(initializeAggregation('covarPopState', 'foo')) on backward Executing query CREATE TABLE atomic.detached (n int) ENGINE=Log on node Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster' TO Disk('backups', '14') SETTINGS replica_num=2 on node1 Skipping covarPop Checking quantileBFloat16Weighted Executing query select hex(initializeAggregation('quantileBFloat16WeightedState', 'foo')) on backward Executing query INSERT INTO atomic.mt1 (n) VALUES (6) on node Skipping quantileBFloat16Weighted Checking quantileExactWeighted Executing query select hex(initializeAggregation('quantileExactWeightedState', 'foo')) on backward Executing query INSERT INTO atomic.mt2 (n) VALUES (6) on node Executing query DROP DATABASE mydb ON CLUSTER 'cluster' SYNC on node1 Skipping quantileExactWeighted Checking quantileTDigestWeighted Executing query select hex(initializeAggregation('quantileTDigestWeightedState', 'foo')) on backward Executing query INSERT INTO atomic.rmt1 (n) VALUES (6) on node Skipping quantileTDigestWeighted Checking quantileExact Executing query select hex(initializeAggregation('quantileExactState', 'foo')) on backward Executing query INSERT INTO atomic.rmt2 (n) VALUES (6) on node Executing query RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM Disk('backups', '14') SETTINGS replica_num_in_backup=2 on node1 Skipping quantileExact Checking stddevPop Executing query select hex(initializeAggregation('stddevPopState', 'foo')) on backward Executing query INSERT INTO atomic.mv1 (n) VALUES (6) on node Skipping stddevPop Checking stddevSamp Executing query select hex(initializeAggregation('stddevSampState', 'foo')) on backward Stderr: node1 Pulling Stderr: node1 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/.env --project-name roottestconcurrentqueriesforallusersrestriction-gw6 --file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/.env --project-name roottestconcurrentqueriesforallusersrestriction-gw6 --file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/docker-compose.yml up -d --no-recreate] Skipping stddevSamp Checking quantileExactLow Executing query select hex(initializeAggregation('quantileExactLowState', 'foo')) on backward Executing query INSERT INTO atomic.mv2 (n) VALUES (6) on node Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/.env --project-name roottestcustomsettings-gw7 --file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/.env --project-name roottestcustomsettings-gw7 --file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/docker-compose.yml up -d --no-recreate] Stderr: node2 Pulling Stderr: node1 Pulling Stderr: node2 Pulled Stderr: node1 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/.env --project-name roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/.env --project-name roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/docker-compose.yml up -d --no-recreate] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/.env --project-name roottestbackuprestorenewshutdownwaitbackup-gw4 --file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/.env --project-name roottestbackuprestorenewshutdownwaitbackup-gw4 --file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/docker-compose.yml up -d --no-recreate] Skipping quantileExactLow Checking groupBitOr Executing query select hex(initializeAggregation('groupBitOrState', 'foo')) on backward Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb.tbl on node1 Executing query INSERT INTO atomic.detached (n) VALUES (6) on node Skipping groupBitOr Checking quantileDeterministic Executing query select hex(initializeAggregation('quantileDeterministicState', 'foo')) on backward Executing query CREATE TABLE `.o r d i n a r y.`.`t. a. b. l. e.` (n int) ENGINE=MergeTree ORDER BY n on node Stderr: Network roottestconcurrentqueriesforallusersrestriction-gw6_default Creating Stderr: Network roottestconcurrentqueriesforallusersrestriction-gw6_default Created Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Creating Executing query SELECT _part, * FROM mydb.tbl ORDER BY x on node1 Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Created Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Starting Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestconcurrentqueriesforallusersrestriction-gw6-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestconcurrentqueriesforallusersrestriction-gw6-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.2... http://localhost:None "GET /v1.46/containers/roottestconcurrentqueriesforallusersrestriction-gw6-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None Skipping quantileDeterministic Aggregate functions: 163, Failed: 0, skipped: 142, passed: 21 [gw5] PASSED test_backward_compatibility/test_functions.py::test_aggregate_states test_backward_compatibility/test_functions.py::test_string_functions Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on upstream http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None Executing query CREATE TABLE lazy.table (n int) ENGINE=Log on node http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None Stderr: Network roottestcustomsettings-gw7_default Creating Stderr: Network roottestcustomsettings-gw7_default Created Stderr: Container roottestcustomsettings-gw7-node-1 Creating Stderr: Container roottestcustomsettings-gw7-node-1 Created Stderr: Container roottestcustomsettings-gw7-node-1 Starting Stderr: Container roottestcustomsettings-gw7-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestcustomsettings-gw7-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestcustomsettings-gw7-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.2.2... http://localhost:None "GET /v1.46/containers/roottestcustomsettings-gw7-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on upstream Executing query CREATE TABLE ordinary.l (n DEFAULT dictGet('other.d1', 'm', toUInt64(3))) ENGINE=Log on node Executing query SELECT _part, * FROM mydb.tbl ORDER BY x on node2 http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None Stderr: Network roottestbackwardcompatibilityselectaggregatealiascolumn-gw1_default Creating Stderr: Network roottestbackwardcompatibilityselectaggregatealiascolumn-gw1_default Created Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Creating Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Creating Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Created Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Created Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Starting Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Starting Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Started Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.5.2... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None Executing query CREATE TABLE other.l (n DEFAULT dictGet('ordinary.d1', 'm', toUInt64(3))) ENGINE=StripeLog on node http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None Executing query SELECT value FROM system.build_options WHERE name = 'CXX_FLAGS' on upstream http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None Executing query SELECT * FROM mydb.tbl ORDER BY x on node2 Stderr: Network roottestbackuprestorenewshutdownwaitbackup-gw4_default Creating Stderr: Network roottestbackuprestorenewshutdownwaitbackup-gw4_default Created Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Creating Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Created Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Starting Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestbackuprestorenewshutdownwaitbackup-gw4-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestbackuprestorenewshutdownwaitbackup-gw4-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.6.2... http://localhost:None "GET /v1.46/containers/roottestbackuprestorenewshutdownwaitbackup-gw4-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None Executing query CREATE TABLE atomic.l (n DEFAULT dictGet('ordinary.d1', 'm', toUInt64(3))) ENGINE=TinyLog on node http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/.env --project-name roottestbackwardcompatibilityfunctions-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/docker-compose.yml stop --timeout 20] [gw5] SKIPPED test_backward_compatibility/test_functions.py::test_string_functions Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None Executing query SELECT count(), sum(n) FROM ordinary.merge on node http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None Executing query SELECT count(), sum(n) FROM other.merge on node http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/d63c5072fb8f6345a2876b44e2cdccae3c8e09e1ae8c094625042e635ff0fd32/json HTTP/1.1" 200 None ClickHouse node1 started Executing query create table nums (number UInt64) ENGINE = MergeTree() order by tuple() on node1 http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None Executing query SELECT count(), sum(n) FROM atomic.merge on node http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/6097cfa6e8f6b39df8ad554bb3ab5a58974956dfd0212ae52f7fc2e8b9b65742/json HTTP/1.1" 200 None ClickHouse node started run container_id:roottestcustomsettings-gw7-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxjdXN0b21fYT5JbnQ2NF8tNTwvY3VzdG9tX2E+CiAgICAgICAgICAgIDxjdXN0b21fYj5VSW50NjRfMTAwMDAwMDAwMDA8L2N1c3RvbV9iPgogICAgICAgICAgICA8Y3VzdG9tX2M+RmxvYXQ2NF8tNDMuMjVlLTE8L2N1c3RvbV9jPgogICAgICAgICAgICA8Y3VzdG9tX2Q+J3NvbWUgdGV4dCc8L2N1c3RvbV9kPgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestcustomsettings-gw7-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxjdXN0b21fYT5JbnQ2NF8tNTwvY3VzdG9tX2E+CiAgICAgICAgICAgIDxjdXN0b21fYj5VSW50NjRfMTAwMDAwMDAwMDA8L2N1c3RvbV9iPgogICAgICAgICAgICA8Y3VzdG9tX2M+RmxvYXQ2NF8tNDMuMjVlLTE8L2N1c3RvbV9jPgogICAgICAgICAgICA8Y3VzdG9tX2Q+J3NvbWUgdGV4dCc8L2N1c3RvbV9kPgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml] Executing query insert into nums values (0), (1) on node1 http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None Executing query SYSTEM RELOAD CONFIG on node http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None Executing query DETACH TABLE ordinary.detached PERMANENTLY on node Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b17e599ee193555d5337a4c85e7eaf8b716b7d4eabe1037cfaac3dd91da2da6a/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.5.3... Executing query select number from nums order by number on node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/873aa1c958dcd34357b266258706fbc19b5f73216468e32a9d1d42c23bc25db7/json HTTP/1.1" 200 None ClickHouse node2 started Executing query create table tab (x UInt64, x_alias UInt64 ALIAS x) engine = Memory on node1 Executing query SELECT getSetting('custom_a') on node http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/flags/convert_ordinary_to_atomic'] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c touch /var/lib/clickhouse/flags/convert_ordinary_to_atomic] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps -C clickhouse] http://localhost:None "GET /v1.46/containers/9e08f3bdd4e4aaad8194f554a1825217dc7a11057b464808f7ac422ea5e4654c/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE TABLE tbl (x UInt64) ENGINE=MergeTree() ORDER BY tuple() PARTITION BY x%5 on node Stdout: PID TTY TIME CMD Stdout: 387 ? 00:00:07 clickhouse run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c pkill clickhouse] Executing query create table tab (x UInt64, x_alias UInt64 ALIAS x) engine = Memory on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query SELECT getSetting('custom_b') on node Stdout:387 Executing query insert into tab values (1) on node1 Executing query INSERT INTO tbl SELECT number FROM numbers(500) on node Executing query select sleep(2) on node1 Executing query SELECT getSetting('custom_c') on node Executing query insert into tab values (1) on node2 test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[database] Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/databases/{database}','{shard}','{replica}') on node1 Executing query select sleep(2) on node1 Executing query SELECT getSetting('custom_d') on node Executing query BACKUP TABLE tbl TO Disk('backups', '3533842410e44d8783116d79c03d0f08') SETTINGS id='3533842410e44d8783116d79c03d0f08' ASYNC on node Executing query select sleep(2) on node1 Executing query select sum(x_alias) from remote('node{1,2}', default, tab) on node1 Executing query CREATE TABLE mydb.tbl(x Int64) ENGINE=ReplicatedMergeTree ORDER BY x on node1 Executing query select sleep(2) on node1 Executing query SELECT status FROM system.backups WHERE id='3533842410e44d8783116d79c03d0f08' on node Command:[docker compose --env-file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/.env --project-name roottestcustomsettings-gw7 --file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/docker-compose.yml stop --timeout 20] [gw7] PASSED test_custom_settings/test.py::test_custom_settings Executing query select sum(x_alias) from remote('node{1,2}', default, tab) on node2 Executing query select sleep(2) on node1 Executing query SELECT count() FROM system.processes WHERE query_kind='Backup' AND query LIKE '%3533842410e44d8783116d79c03d0f08%' on node Executing query select number from remote('node1', 'default', 'nums') on node1 Executing query INSERT INTO mydb.tbl VALUES (-3) on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query drop table tab on node1 Stdout:387 Executing query select sleep(2) on node1 Executing query select number from remote('node1', 'default', 'nums') on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps -C clickhouse] Executing query INSERT INTO mydb.tbl VALUES (1) on node1 Executing query drop table tab on node2 Stdout: PID TTY TIME CMD Stdout: 8 ? 00:00:01 clickhouse run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c pkill clickhouse] Executing query select sleep(2) on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query select sleep(2) on node1 Executing query select number from remote('node1', 'default', 'nums') on node1 Executing query INSERT INTO mydb.tbl VALUES (10) on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/.env --project-name roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/docker-compose.yml stop --timeout 20] [gw1] PASSED test_backward_compatibility/test_select_aggregate_alias_column.py::test_select_aggregate_alias_column Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query select sleep(2) on node1 Executing query select number from remote('node1', 'default', 'nums') on node1 Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster' TO Disk('backups', '15') on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/.env --project-name roottestconcurrentqueriesforallusersrestriction-gw6 --file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/docker-compose.yml stop --timeout 20] [gw6] PASSED test_concurrent_queries_for_all_users_restriction/test.py::test_exception_message run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:387 Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Stopping Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Stopping Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Stopped Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query select sleep(2) on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/.env --project-name roottestbackwardcompatibilityfunctions-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/upstream/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-functions-0-gw5/backward/docker-compose.yml down --volumes] Executing query RESTORE DATABASE mydb AS mydb2 ON CLUSTER 'cluster' FROM Disk('backups', '15') on node1 Executing query select sleep(2) on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query INSERT INTO mydb.tbl VALUES (2) on node1 Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Stopping Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Stopping Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Stopped Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Removing Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Stopped Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Removing Stderr: Container roottestbackwardcompatibilityfunctions-gw5-upstream-1 Removed Stderr: Container roottestbackwardcompatibilityfunctions-gw5-backward-1 Removed Stderr: Network roottestbackwardcompatibilityfunctions-gw5_default Removing Stderr: Network roottestbackwardcompatibilityfunctions-gw5_default Removed Cleanup called Docker networks for project roottestbackwardcompatibilityfunctions-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityfunctions-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityfunctions-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityfunctions-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityfunctions-gw5 Trying to prune unused networks... Executing query select sleep(2) on node1 Trying to prune unused images... Command:[docker image prune -f] Executing query select sleep(2) on node1 Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Running tests in /ClickHouse/tests/integration/test_backward_compatibility/test_cte_distributed.py test_backward_compatibility/test_cte_distributed.py::test_cte_distributed Cluster start called. is_up=False Docker networks for project roottestbackwardcompatibilityctedistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityctedistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityctedistributed-gw5 are DRIVER VOLUME NAME Cleanup called Executing query SYSTEM SYNC DATABASE REPLICA ON CLUSTER 'cluster' mydb2 on node1 Docker networks for project roottestbackwardcompatibilityctedistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityctedistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityctedistributed-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityctedistributed-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityctedistributed-gw5 Trying to prune unused networks... Executing query select sleep(2) on node1 Trying to prune unused images... Command:[docker image prune -f] Executing query select sleep(2) on node1 Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/.env --project-name roottestbackwardcompatibilityctedistributed-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/docker-compose.yml pull] Stdout:387 Stdout:1191 Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb2.tbl on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Stdout:8 Executing query SELECT * FROM mydb.tbl ORDER BY x on node1 Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query select sleep(2) on node1 Executing query SELECT * FROM mydb2.tbl ORDER BY x on node1 Executing query SELECT * FROM mydb2.tbl ORDER BY x on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[database] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/02856ef6d13cef61eecfde6c17166c3199c30fdf984c9d7007ac702d9ec8a641/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/02856ef6d13cef61eecfde6c17166c3199c30fdf984c9d7007ac702d9ec8a641/json HTTP/1.1" 200 586 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Stopping Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/.env --project-name roottestconcurrentqueriesforallusersrestriction-gw6 --file /ClickHouse/tests/integration/test_concurrent_queries_for_all_users_restriction/_instances-0-gw6/node1/docker-compose.yml down --volumes] Stderr: Container roottestcustomsettings-gw7-node-1 Stopping Stderr: Container roottestcustomsettings-gw7-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/.env --project-name roottestcustomsettings-gw7 --file /ClickHouse/tests/integration/test_custom_settings/_instances-0-gw7/node/docker-compose.yml down --volumes] Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Stopping Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Stopping Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Stopped Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/.env --project-name roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-select_aggregate_alias_column-0-gw1/node2/docker-compose.yml down --volumes] Stdout:1228 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1228 Executing query select 20 on node Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Stopping Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Stopped Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Removing Stderr: Container roottestconcurrentqueriesforallusersrestriction-gw6-node1-1 Removed Stderr: Network roottestconcurrentqueriesforallusersrestriction-gw6_default Removing Stderr: Network roottestconcurrentqueriesforallusersrestriction-gw6_default Removed Cleanup called Docker networks for project roottestconcurrentqueriesforallusersrestriction-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestconcurrentqueriesforallusersrestriction-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Docker volumes for project roottestconcurrentqueriesforallusersrestriction-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestconcurrentqueriesforallusersrestriction-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Unstopped containers: {} No running containers for project: roottestconcurrentqueriesforallusersrestriction-gw6 Trying to prune unused networks... Stdout:8 Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[uuid] Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/databases/{uuid}','{shard}','{replica}') on node1 Stderr: Container roottestcustomsettings-gw7-node-1 Stopping Stderr: Container roottestcustomsettings-gw7-node-1 Stopped Stderr: Container roottestcustomsettings-gw7-node-1 Removing Stderr: Container roottestcustomsettings-gw7-node-1 Removed Stderr: Network roottestcustomsettings-gw7_default Removing Stderr: Network roottestcustomsettings-gw7_default Removed Cleanup called Docker networks for project roottestcustomsettings-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcustomsettings-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcustomsettings-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcustomsettings-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcustomsettings-gw7 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Executing query CREATE TABLE mydb.tbl(x Int64) ENGINE=ReplicatedMergeTree ORDER BY x on node1 Executing query select 20 on node Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Stopping Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Stopping Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Stopped Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Removing Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Stopped Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Removing Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node2-1 Removed Stderr: Container roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-node1-1 Removed Stderr: Network roottestbackwardcompatibilityselectaggregatealiascolumn-gw1_default Removing Stderr: Network roottestbackwardcompatibilityselectaggregatealiascolumn-gw1_default Removed Cleanup called Executing query INSERT INTO mydb.tbl VALUES (-3) on node1 Docker networks for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityselectaggregatealiascolumn-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityselectaggregatealiascolumn-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_cancel_freeze/test.py::test_cancel_backup Running tests in /ClickHouse/tests/integration/test_cancel_freeze/test.py Cluster start called. is_up=False Stdout:8 Docker networks for project roottestcancelfreeze-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcancelfreeze-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query INSERT INTO mydb.tbl VALUES (1) on node1 Docker volumes for project roottestcancelfreeze-gw1 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestcancelfreeze-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcancelfreeze-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcancelfreeze-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcancelfreeze-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcancelfreeze-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query select 20 on node Stdout:1 Volumes pruned: 1 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/database Setup logs dir /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/.env --project-name roottestcancelfreeze-gw1 --file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/docker-compose.yml pull] Executing query INSERT INTO mydb.tbl VALUES (10) on node1 Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster' TO Disk('backups', '16') on node1 Executing query select 20 on node Executing query RESTORE DATABASE mydb AS mydb2 ON CLUSTER 'cluster' FROM Disk('backups', '16') on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query select 20 on node Executing query INSERT INTO mydb.tbl VALUES (2) on node1 Executing query SYSTEM SYNC DATABASE REPLICA ON CLUSTER 'cluster' mydb2 on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' mydb2.tbl on node1 Executing query select 20 on node Executing query SELECT * FROM mydb.tbl ORDER BY x on node1 Executing query SELECT * FROM mydb2.tbl ORDER BY x on node1 Executing query SELECT * FROM mydb2.tbl ORDER BY x on node2 Executing query select 20 on node Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[uuid] run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query select 20 on node Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query select 20 on node Stdout:8 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 test_backup_restore_on_cluster/test.py::test_replicated_table Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8, y String) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Executing query select 20 on node Executing query INSERT INTO tbl VALUES (1, 'Don''t') on node1 Executing query INSERT INTO tbl VALUES (2, 'count') on node2 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query INSERT INTO tbl SETTINGS async_insert=true VALUES (3, 'your') on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Current start attempt failed. Will kill 1228 just in case. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 1228'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c kill -9 1228] Executing query INSERT INTO tbl SETTINGS async_insert=true VALUES (4, 'chickens') on node2 Stderr:bash: line 0: kill: (1228) - No such process Exitcode:1 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '17') SETTINGS replica_num=1 on node1 http://localhost:None "POST /v1.46/exec/b31ba41b7e0d3226d79d06da0d2833e171d2b515addb46110de2c2528318ff20/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/b31ba41b7e0d3226d79d06da0d2833e171d2b515addb46110de2c2528318ff20/json HTTP/1.1" 200 586 Stdout:8 Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '17') on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2015 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Stderr: node1 Pulling Stderr: node2 Pulling Stderr: node1 Pulled Stderr: node2 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/.env --project-name roottestbackwardcompatibilityctedistributed-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/.env --project-name roottestbackwardcompatibilityctedistributed-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/docker-compose.yml up -d --no-recreate] Stdout:2015 Executing query select 20 on node Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 Executing query SELECT * FROM tbl ORDER BY x on node2 Stderr: Network roottestbackwardcompatibilityctedistributed-gw5_default Creating Stderr: Network roottestbackwardcompatibilityctedistributed-gw5_default Created Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Creating Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Creating Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Created Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Created Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Starting Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Starting Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Started Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityctedistributed-gw5-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityctedistributed-gw5-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.2... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityctedistributed-gw5-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None Executing query select 20 on node http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None Executing query SELECT * FROM tbl ORDER BY x on node1 http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None Stdout:8 http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/.env --project-name roottestcancelfreeze-gw1 --file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/.env --project-name roottestcancelfreeze-gw1 --file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/docker-compose.yml up -d --no-recreate] Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None Executing query select 20 on node http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None Stderr: Network roottestcancelfreeze-gw1_default Creating Stderr: Network roottestcancelfreeze-gw1_default Created Stderr: Container roottestcancelfreeze-gw1-node-1 Creating Stderr: Container roottestcancelfreeze-gw1-node-1 Created Stderr: Container roottestcancelfreeze-gw1-node-1 Starting Stderr: Container roottestcancelfreeze-gw1-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestcancelfreeze-gw1-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestcancelfreeze-gw1-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.2.2... http://localhost:None "GET /v1.46/containers/roottestcancelfreeze-gw1-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c869c4c71f8e42be21dd79f736d1e91cd2266f7887900445d3dda1119e249ec6/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityctedistributed-gw5-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityctedistributed-gw5-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.1.3... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityctedistributed-gw5-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0834013a11d63c1a7af6d4febe0d39b41c1e70ae2c362ae1661ed23ba9cebbb1/json HTTP/1.1" 200 None ClickHouse node2 started Executing query WITH quantile(0.05)(cnt) as p05, quantile(0.95)(cnt) as p95, p95 - p05 as inter_percentile_range SELECT sum(cnt) as total_requests, count() as data_points, inter_percentile_range FROM ( SELECT count() as cnt FROM remote('node{1,2}', numbers(10)) GROUP BY number ) on node2 http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None Executing query select 20 on node Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None Stdout:8 http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None Executing query WITH quantile(0.05)(cnt) as p05, quantile(0.95)(cnt) as p95, p95 - p05 as inter_percentile_range SELECT sum(cnt) as total_requests, count() as data_points, inter_percentile_range FROM ( SELECT count() as cnt FROM remote('node{1,2}', numbers(10)) GROUP BY number ) on node1 http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None test_backup_restore_on_cluster/test.py::test_replicated_table_after_alters Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x Int32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/.env --project-name roottestbackwardcompatibilityctedistributed-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/docker-compose.yml stop --timeout 20] [gw5] PASSED test_backward_compatibility/test_cte_distributed.py::test_cte_distributed http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None Executing query INSERT INTO tbl VALUES (1) on node1 http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None Executing query select 20 on node http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None Executing query ALTER TABLE tbl ADD COLUMN y Int32 on node1 http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/d67b194a206c9a2f996ee3596b2db292b5204ff9847fd67369091c9e4f6eb167/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE TABLE tbl (x UInt32, y UInt32) ENGINE=MergeTree() PARTITION BY (x%20000) ORDER BY x on node Executing query INSERT INTO tbl VALUES (2, 20) on node1 Stdout:8 Executing query INSERT INTO tbl SELECT number, number FROM numbers(20000) SETTINGS max_partitions_per_insert_block=20000 on node Executing query ALTER TABLE tbl ADD COLUMN z Int32 on node1 Executing query select 20 on node Executing query INSERT INTO tbl VALUES (3, 30, 300) on node1 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '18') on node1 Executing query select 20 on node Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '18') on node1 Executing query select 20 on node Executing query OPTIMIZE TABLE tbl FINAL on node1 Executing query SELECT * FROM tbl ORDER BY x on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query SELECT * FROM tbl ORDER BY x on node2 Executing query select 20 on node Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_after_alters Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query select 20 on node run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Stopping Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Stopping Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Stopped Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/.env --project-name roottestbackwardcompatibilityctedistributed-gw5 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-cte_distributed-0-gw5/node2/docker-compose.yml down --volumes] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Current start attempt failed. Will kill 2015 just in case. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 2015'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c kill -9 2015] Stderr:bash: line 0: kill: (2015) - No such process Exitcode:1 test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Executing query INSERT INTO tbl VALUES (111) on node1 Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Stopping Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Stopping Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Stopped Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Removing Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Stopped Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Removing Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node1-1 Removed Stderr: Container roottestbackwardcompatibilityctedistributed-gw5-node2-1 Removed Stderr: Network roottestbackwardcompatibilityctedistributed-gw5_default Removing Stderr: Network roottestbackwardcompatibilityctedistributed-gw5_default Removed Cleanup called Docker networks for project roottestbackwardcompatibilityctedistributed-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityctedistributed-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityctedistributed-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityctedistributed-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityctedistributed-gw5 Trying to prune unused networks... run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query INSERT INTO tbl VALUES (222) on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1 Volumes pruned: 1 Stdout:8 No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/7ddb58c938c23d3fb206f50b95d76cac7534c283061f3702cfa714df56de10c0/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/7ddb58c938c23d3fb206f50b95d76cac7534c283061f3702cfa714df56de10c0/json HTTP/1.1" 200 586 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '19') on node1 Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster3' FROM Disk('backups', '19') on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2792 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2792 Executing query select 20 on node Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster3' tbl on node1 Executing query SELECT * FROM tbl ORDER BY x on node1 Executing query select 20 on node Executing query SELECT * FROM tbl ORDER BY x on node2 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT * FROM tbl ORDER BY x on node3 Stdout:8 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster Executing query select 20 on node Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query select 20 on node Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Executing query select 20 on node Executing query INSERT INTO tbl VALUES (111) on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query INSERT INTO tbl VALUES (222) on node2 Stdout:8 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '20') on node1 Executing query select 20 on node Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster1' FROM Disk('backups', '20') on node1 Executing query select 20 on node run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query SELECT * FROM tbl ORDER BY x on node1 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster Executing query select 20 on node Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query select 20 on node Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query select 20 on node test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def Executing query CREATE TABLE tbl (x UInt8, y String) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY tuple() on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query CREATE TABLE tbl (x UInt8, y String) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY tuple() on node2 Executing query SYSTEM STOP REPLICATION QUEUES tbl on node2 Executing query ALTER TABLE tbl MODIFY COLUMN x String on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Current start attempt failed. Will kill 2792 just in case. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 2792'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c kill -9 2792] Stderr:bash: line 0: kill: (2792) - No such process Exitcode:1 Executing query SELECT name, type FROM system.columns WHERE database='default' AND table='tbl' on node1 Executing query SELECT name, type FROM system.columns WHERE database='default' AND table='tbl' on node2 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '21') on node2 No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/23740daa574053645f67f918a459de636a09e0d298e2eedd67712654feec17d4/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/23740daa574053645f67f918a459de636a09e0d298e2eedd67712654feec17d4/json HTTP/1.1" 200 586 Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '21') SETTINGS replica_num_in_backup=1 on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:3555 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:3555 Executing query select 20 on node Executing query SELECT name, type FROM system.columns WHERE database='default' AND table='tbl' on node1 Executing query SELECT name, type FROM system.columns WHERE database='default' AND table='tbl' on node2 Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query select 20 on node Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '21') SETTINGS replica_num_in_backup=2 on node2 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackuprestorenewshutdownwaitbackup-gw4-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/198c36c156738f8aabb9b37a6b073ab261f955a5630776e9855cf139aadb3004/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/198c36c156738f8aabb9b37a6b073ab261f955a5630776e9855cf139aadb3004/json HTTP/1.1" 200 586 Executing query SELECT name, type FROM system.columns WHERE database='default' AND table='tbl' on node1 Executing query select 20 on node Executing query SELECT name, type FROM system.columns WHERE database='default' AND table='tbl' on node2 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query select 20 on node Stdout:1073 Clickhouse process running. run container_id:roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1073 Executing query select 20 on node Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query select 20 on node Executing query select 20 on node test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Executing query SELECT status FROM system.backup_log WHERE id='3533842410e44d8783116d79c03d0f08' ORDER BY status on node Executing query DROP TABLE tbl SYNC on node Executing query INSERT INTO tbl VALUES (111) on node1 Executing query select 20 on node Executing query RESTORE TABLE tbl FROM Disk('backups', '3533842410e44d8783116d79c03d0f08') on node Executing query INSERT INTO tbl VALUES (222) on node2 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 Executing query SYSTEM STOP REPLICATED SENDS ON CLUSTER 'cluster' tbl on node1 Executing query select 20 on node Executing query INSERT INTO tbl VALUES (333) on node1 Executing query INSERT INTO tbl VALUES (444) on node2 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '22') on node1 Executing query select 20 on node Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query select 20 on node Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '22') on node1 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 Executing query select 20 on node Executing query SELECT * FROM tbl ORDER BY x on node1 Executing query SELECT * FROM tbl ORDER BY x on node2 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Current start attempt failed. Will kill 3555 just in case. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 3555'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c kill -9 3555] Stderr:bash: line 0: kill: (3555) - No such process Exitcode:1 Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/120de20cc80566efada2e26dfefc931a5a8d9fe3df57f874fc9a817661d5eaee/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/120de20cc80566efada2e26dfefc931a5a8d9fe3df57f874fc9a817661d5eaee/json HTTP/1.1" 200 586 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4326 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4326 Executing query select 20 on node Executing query SYSTEM STOP MERGES ON CLUSTER 'cluster' tbl on node1 Executing query INSERT INTO tbl VALUES (111) on node1 Executing query INSERT INTO tbl VALUES (222) on node1 Executing query select 20 on node Executing query SYSTEM SYNC REPLICA tbl on node2 Executing query SYSTEM START MERGES tbl on node2 Executing query OPTIMIZE TABLE tbl FINAL on node2 Executing query select 20 on node Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '23') on node1 Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query select 20 on node Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '23') on node1 Executing query select 20 on node Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 Executing query SELECT * FROM tbl ORDER BY x on node1 Executing query DROP TABLE IF EXISTS tbl SYNC on node [gw4] PASSED test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup Executing query SELECT * FROM tbl ORDER BY x on node2 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/.env --project-name roottestbackuprestorenewshutdownwaitbackup-gw4 --file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/docker-compose.yml stop --timeout 20] Executing query select 20 on node Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Stopping Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/.env --project-name roottestbackuprestorenewshutdownwaitbackup-gw4 --file /ClickHouse/tests/integration/test_backup_restore_new/_instances-shutdown_wait_backup-0-gw4/node/docker-compose.yml down --volumes] Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query select 20 on node Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Stopping Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Stopped Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Removing Stderr: Container roottestbackuprestorenewshutdownwaitbackup-gw4-node-1 Removed Stderr: Network roottestbackuprestorenewshutdownwaitbackup-gw4_default Removing Stderr: Network roottestbackuprestorenewshutdownwaitbackup-gw4_default Removed Cleanup called Docker networks for project roottestbackuprestorenewshutdownwaitbackup-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackuprestorenewshutdownwaitbackup-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackuprestorenewshutdownwaitbackup-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackuprestorenewshutdownwaitbackup-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackuprestorenewshutdownwaitbackup-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 test_cluster_discovery/test_password.py::test_connect_with_password Running tests in /ClickHouse/tests/integration/test_cluster_discovery/test_password.py Cluster start called. is_up=False Docker networks for project roottestclusterdiscoverypassword-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestclusterdiscoverypassword-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestclusterdiscoverypassword-gw4 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestclusterdiscoverypassword-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestclusterdiscoverypassword-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestclusterdiscoverypassword-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestclusterdiscoverypassword-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestclusterdiscoverypassword-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query select 20 on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node0 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_cluster_discovery/config/config_with_pwd.xml', '/ClickHouse/tests/integration/test_cluster_discovery/config/config_with_secret1.xml'] to /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/configs/config.d Setup database dir /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/database Setup logs dir /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_cluster_discovery/config/config_with_pwd.xml', '/ClickHouse/tests/integration/test_cluster_discovery/config/config_with_secret2.xml'] to /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/database Setup logs dir /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/.env --project-name roottestclusterdiscoverypassword-gw4 --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/docker-compose.yml pull] test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8, y String) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{uuid}','{replica}')ORDER BY x on node1 Executing query INSERT INTO tbl VALUES (1, 'AA') on node1 Executing query select 20 on node Executing query INSERT INTO tbl VALUES (2, 'BB') on node2 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '24') on node1 Executing query select 20 on node Executing query RESTORE TABLE tbl AS tbl2 ON CLUSTER 'cluster' FROM Disk('backups', '24') on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Current start attempt failed. Will kill 4326 just in case. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 4326'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c kill -9 4326] Stderr:bash: line 0: kill: (4326) - No such process Exitcode:1 Executing query INSERT INTO tbl2 VALUES (3, 'CC') on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node1 No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/c9a0a48477642ac808321ebe3a7d683d0a8c2248f3f92ffb4c872fd391980076/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/c9a0a48477642ac808321ebe3a7d683d0a8c2248f3f92ffb4c872fd391980076/json HTTP/1.1" 200 586 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl2 on node1 Executing query SELECT * FROM tbl ORDER BY x on node1 Executing query SELECT * FROM tbl2 ORDER BY x on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT * FROM tbl ORDER BY x on node2 Stdout:5105 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5105 Executing query select 20 on node Executing query SELECT * FROM tbl2 ORDER BY x on node2 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query select 20 on node Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query select 20 on node Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 test_backup_restore_on_cluster/test.py::test_required_privileges Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Executing query select 20 on node Executing query INSERT INTO tbl VALUES (100) on node1 Executing query CREATE USER u1 on node1 Executing query GRANT CLUSTER ON *.* TO u1 on node1 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '25') on node1 Executing query select 20 on node Executing query GRANT BACKUP ON tbl TO u1 on node1 Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '25') on node1 Executing query select 20 on node Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query select 20 on node Executing query RESTORE TABLE tbl AS tbl2 ON CLUSTER 'cluster' FROM Disk('backups', '25') on node1 Executing query GRANT INSERT, CREATE TABLE ON tbl2 TO u1 on node1 Executing query RESTORE TABLE tbl AS tbl2 ON CLUSTER 'cluster' FROM Disk('backups', '25') on node1 Executing query select 20 on node Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl2 on node2 Executing query SELECT * FROM tbl2 on node2 Executing query select 20 on node Executing query DROP TABLE tbl2 ON CLUSTER 'cluster' SYNC on node1 Executing query REVOKE ALL FROM u1 on node1 Executing query select 20 on node Executing query RESTORE ALL ON CLUSTER 'cluster' FROM Disk('backups', '25') on node1 Executing query GRANT INSERT, CREATE TABLE ON tbl TO u1 on node1 Executing query RESTORE ALL ON CLUSTER 'cluster' FROM Disk('backups', '25') on node1 Stderr: zoo2 Skipped - Image is already being pulled by node0 Stderr: zoo3 Skipped - Image is already being pulled by node0 Stderr: zoo1 Skipped - Image is already being pulled by node0 Stderr: node1 Skipped - Image is already being pulled by node0 Stderr: node0 Pulling Stderr: node0 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper1/log', '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper1/config', '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper1/coordination', '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper2/log', '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper2/config', '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper2/coordination', '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper3/log', '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper3/config', '/ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/keeper3/coordination'] Command:[docker compose --project-name roottestclusterdiscoverypassword-gw4 --env-file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Current start attempt failed. Will kill 5105 just in case. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 5105'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c kill -9 5105] Stderr:bash: line 0: kill: (5105) - No such process Exitcode:1 Executing query SYSTEM SYNC REPLICA ON CLUSTER 'cluster' tbl on node2 Executing query SELECT * FROM tbl on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_required_privileges http://localhost:None "POST /v1.46/exec/80cbe78eb65e702acfd2a87a7756ec672850ed270a36bbd2020ef9c08f89e1ea/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/80cbe78eb65e702acfd2a87a7756ec672850ed270a36bbd2020ef9c08f89e1ea/json HTTP/1.1" 200 586 Stderr:time="2025-04-02T02:54:56Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestclusterdiscoverypassword-gw4_default Creating Stderr: Network roottestclusterdiscoverypassword-gw4_default Created Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Creating Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Creating Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Creating Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Created Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Created Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Created Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Starting Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Starting Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Starting Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Started Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Started Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Started Stderr:time="2025-04-02T02:54:57Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:54:57Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Stdout:5868 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5868 Executing query select 20 on node Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query select 20 on node test_backup_restore_on_cluster/test.py::test_shutdown_waits_for_backup Executing query CREATE TABLE tbl ON CLUSTER 'cluster' (x UInt8) ENGINE=ReplicatedMergeTree('/clickhouse/tables/tbl/', '{replica}')ORDER BY x on node1 Executing query INSERT INTO tbl VALUES (3) on node1 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query INSERT INTO tbl VALUES (5) on node2 Executing query select 20 on node Executing query BACKUP TABLE tbl ON CLUSTER 'cluster' TO Disk('backups', '26') ASYNC on node1 run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 8 ? 00:00:31 clickhouse run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestbackuprestoreoncluster-gw2-node2-1 bash -c pkill clickhouse] run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query select 20 on node Executing query select 20 on node Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query select 20 on node run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query select 20 on node Executing query select 20 on node run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query select 20 on node Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/.env --project-name roottestclusterdiscoverypassword-gw4 --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/.env --project-name roottestclusterdiscoverypassword-gw4 --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/docker-compose.yml up -d --no-recreate] run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query select 20 on node Stdout:8 Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Running Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Running Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Running Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Creating Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Creating Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Created Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Created Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Starting Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Starting Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Started Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Started ClickHouse instance created get_instance_ip instance_name=node0 http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-node0-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node0 http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-node0-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node0, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-node0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None Current start attempt failed. Will kill 5868 just in case. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 5868'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c kill -9 5868] Stderr:bash: line 0: kill: (5868) - No such process Exitcode:1 http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "POST /v1.46/exec/02c419d9401ffc214ea222212683d964bea6bfa5363c5379215d43904b844aea/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/02c419d9401ffc214ea222212683d964bea6bfa5363c5379215d43904b844aea/json HTTP/1.1" 200 586 http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/24283fbdbac7a8eeb905a058346977794c2798d2325e5c926e2b3282d2f144f7/json HTTP/1.1" 200 None ClickHouse node0 started get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.6... http://localhost:None "GET /v1.46/containers/roottestclusterdiscoverypassword-gw4-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/464552eedd043a3d1dc86012b78cfba25d09e12624a1af82ad7531d309ba56fe/json HTTP/1.1" 200 None ClickHouse node1 started Executing query SELECT count() FROM system.clusters WHERE cluster = 'test_auto_cluster_with_pwd' on node0 Executing query SELECT count() FROM system.clusters WHERE cluster = 'test_auto_cluster_with_pwd' on node1 run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query SELECT sum(number) FROM clusterAllReplicas('test_auto_cluster_with_pwd', numbers(3)) GROUP BY hostname() on node0 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm /var/lib/clickhouse/flags/convert_ordinary_to_atomic'] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c rm /var/lib/clickhouse/flags/convert_ordinary_to_atomic] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6628 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6628 Executing query select 20 on node Executing query SELECT sum(number) FROM clusterAllReplicas('test_auto_cluster_with_wrong_pwd', numbers(3)) GROUP BY hostname() on node0 Executing query SELECT sum(number) FROM clusterAllReplicas('test_auto_cluster_with_secret', numbers(3)) GROUP BY hostname() on node0 Executing query SELECT sum(number) FROM clusterAllReplicas('test_auto_cluster_with_wrong_secret', numbers(3)) GROUP BY hostname() on node0 run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query select 20 on node Stdout:8 Command:[docker compose --env-file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/.env --project-name roottestclusterdiscoverypassword-gw4 --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/docker-compose.yml stop --timeout 20] [gw4] PASSED test_cluster_discovery/test_password.py::test_connect_with_password Executing query select 20 on node Executing query SELECT engine FROM system.databases where name='ordinary' on node run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Executing query ATTACH TABLE ordinary.detached on node run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'touch /var/lib/clickhouse/flags/convert_ordinary_to_atomic'] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c touch /var/lib/clickhouse/flags/convert_ordinary_to_atomic] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 6628 ? 00:00:06 clickhouse run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c pkill clickhouse] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6628 Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/.env --project-name roottestclusterdiscoverypassword-gw4 --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_cluster_discovery/_instances-password-0-gw4/node1/docker-compose.yml down --volumes] run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Removing Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Removing Stderr: Container roottestclusterdiscoverypassword-gw4-node1-1 Removed Stderr: Container roottestclusterdiscoverypassword-gw4-node0-1 Removed Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Stopping Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Removing Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Removing Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Stopped Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Removing Stderr: Container roottestclusterdiscoverypassword-gw4-zoo3-1 Removed Stderr: Container roottestclusterdiscoverypassword-gw4-zoo2-1 Removed Stderr: Container roottestclusterdiscoverypassword-gw4-zoo1-1 Removed Stderr: Network roottestclusterdiscoverypassword-gw4_default Removing Stderr: Network roottestclusterdiscoverypassword-gw4_default Removed Cleanup called Docker networks for project roottestclusterdiscoverypassword-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestclusterdiscoverypassword-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestclusterdiscoverypassword-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestclusterdiscoverypassword-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestclusterdiscoverypassword-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6628 run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT status FROM system.backups WHERE id='cedb0024-d5f2-4374-bced-c2100d848b20' AND status == 'CREATING_BACKUP' on node1 Executing query SELECT status FROM system.backups WHERE id='cedb0024-d5f2-4374-bced-c2100d848b20' on node1 run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackuprestoreoncluster-gw2-node2-1/exec HTTP/1.1" 201 74 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "POST /v1.46/exec/d8ea068c183333e092dddb06e0d0a565226ea3a30e521c7cf3d6d51c18a59b98/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/d8ea068c183333e092dddb06e0d0a565226ea3a30e521c7cf3d6d51c18a59b98/json HTTP/1.1" 200 586 Stdout:6628 run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:880 Clickhouse process running. run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6628 Stdout:880 Executing query select 20 on node2 Executing query select 20 on node2 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityconvertordinary-gw9-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/7e6da237a7f5dec3f22d1ec60a64df4ef7f9c40d9964d5f6adbe10b24a03b30d/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/7e6da237a7f5dec3f22d1ec60a64df4ef7f9c40d9964d5f6adbe10b24a03b30d/json HTTP/1.1" 200 586 Connection dropped: socket connection error: None Failed connecting to Zookeeper within the connection retry policy. Connection dropped: socket connection error: None Failed connecting to Zookeeper within the connection retry policy. Executing query select 20 on node2 Executing query DROP TABLE tbl ON CLUSTER 'cluster' SYNC on node1 Executing query RESTORE TABLE tbl ON CLUSTER 'cluster' FROM Disk('backups', '26') on node1 run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7459 Clickhouse process running. run container_id:roottestbackwardcompatibilityconvertordinary-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityconvertordinary-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7459 Executing query select 20 on node Executing query SYSTEM SYNC REPLICA tbl on node1 Executing query SELECT * FROM tbl ORDER BY x on node1 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_shutdown_waits_for_backup Executing query select 20 on node Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query select 20 on node Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query SELECT name FROM system.databases WHERE engine='Atomic' ORDER BY name on node Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query SELECT engine FROM system.databases WHERE name IN ('mem', 'lazy') ORDER BY name on node test_backup_restore_on_cluster/test.py::test_system_functions Executing query CREATE FUNCTION linear_equation AS (x, k, b) -> k*x + b; on node1 Executing query SHOW TABLES FROM `.o r d i n a r y.` on node Executing query CREATE FUNCTION parity_str AS (n) -> if(n % 2, 'odd', 'even'); on node1 Executing query SELECT uuid FROM system.tables WHERE name='tbl' on node Executing query SHOW TABLES FROM lazy on node Executing query BACKUP TABLE system.functions ON CLUSTER 'cluster' TO Disk('backups', '27') on node1 Executing query ALTER TABLE tbl FREEZE WITH NAME 'test' on node run container_id:roottestcancelfreeze-gw1-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls /var/lib/clickhouse/shadow/'] Command:[docker exec roottestcancelfreeze-gw1-node-1 bash -c ls /var/lib/clickhouse/shadow/] Stderr:ls: cannot access '/var/lib/clickhouse/shadow/': No such file or directory Exitcode:2 run container_id:roottestcancelfreeze-gw1-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls /var/lib/clickhouse/shadow/'] Command:[docker exec roottestcancelfreeze-gw1-node-1 bash -c ls /var/lib/clickhouse/shadow/] Stderr:ls: cannot access '/var/lib/clickhouse/shadow/': No such file or directory Exitcode:2 run container_id:roottestcancelfreeze-gw1-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls /var/lib/clickhouse/shadow/'] Command:[docker exec roottestcancelfreeze-gw1-node-1 bash -c ls /var/lib/clickhouse/shadow/] Executing query SHOW TABLES FROM ordinary NOT LIKE '%inner%' on node Stdout:increment.txt Stdout:test Executing query SELECT count() FROM system.processes WHERE query_kind == 'Alter' AND query LIKE '%FREEZE%' on node Executing query SHOW TABLES FROM other NOT LIKE '%inner%' on node Executing query KILL QUERY WHERE query_kind == 'Alter' AND query LIKE '%FREEZE%' SYNC on node Executing query DROP FUNCTION linear_equation on node1 Executing query SHOW TABLES FROM atomic NOT LIKE '%inner%' on node Executing query DROP FUNCTION parity_str on node1 run container_id:roottestcancelfreeze-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'ls /var/lib/clickhouse/shadow/test/store/adf/adfe9f4d-03c0-4421-8b93-65ab287c8a9b'] Command:[docker exec roottestcancelfreeze-gw1-node-1 bash -c ls /var/lib/clickhouse/shadow/test/store/adf/adfe9f4d-03c0-4421-8b93-65ab287c8a9b] Executing query SELECT count(), sum(n) FROM ordinary.merge on node Executing query SELECT name FROM system.functions WHERE name='parity_str' on node2 Stdout:0_1_1_0 Stdout:10000_10001_10001_0 Stdout:1000_1001_1001_0 Stdout:10001_10002_10002_0 Stdout:10002_10003_10003_0 Stdout:10003_10004_10004_0 Stdout:10004_10005_10005_0 Stdout:10005_10006_10006_0 Stdout:10006_10007_10007_0 Stdout:10007_10008_10008_0 Stdout:10008_10009_10009_0 Stdout:10009_10010_10010_0 Stdout:10010_10011_10011_0 Stdout:100_101_101_0 Stdout:1001_1002_1002_0 Stdout:10011_10012_10012_0 Stdout:10012_10013_10013_0 Stdout:10013_10014_10014_0 Stdout:10014_10015_10015_0 Stdout:10015_10016_10016_0 Stdout:10016_10017_10017_0 Stdout:10017_10018_10018_0 Stdout:10018_10019_10019_0 Stdout:10019_10020_10020_0 Stdout:10020_10021_10021_0 Stdout:1002_1003_1003_0 Stdout:10021_10022_10022_0 Stdout:10022_10023_10023_0 Stdout:10023_10024_10024_0 Stdout:10024_10025_10025_0 Stdout:10025_10026_10026_0 Stdout:10026_10027_10027_0 Stdout:10027_10028_10028_0 Stdout:10028_10029_10029_0 Stdout:10029_10030_10030_0 Stdout:10030_10031_10031_0 Stdout:1003_1004_1004_0 Stdout:10031_10032_10032_0 Stdout:10032_10033_10033_0 Stdout:10033_10034_10034_0 Stdout:10034_10035_10035_0 Stdout:10035_10036_10036_0 Stdout:10036_10037_10037_0 Stdout:10037_10038_10038_0 Stdout:10038_10039_10039_0 Stdout:10039_10040_10040_0 Stdout:10040_10041_10041_0 Stdout:1004_1005_1005_0 Stdout:10041_10042_10042_0 Stdout:10042_10043_10043_0 Stdout:10043_10044_10044_0 Stdout:10044_10045_10045_0 Stdout:10045_10046_10046_0 Stdout:10046_10047_10047_0 Stdout:10047_10048_10048_0 Stdout:10048_10049_10049_0 Stdout:10049_10050_10050_0 Stdout:10050_10051_10051_0 Stdout:1005_1006_1006_0 Stdout:10051_10052_10052_0 Stdout:10052_10053_10053_0 Stdout:10053_10054_10054_0 Stdout:10054_10055_10055_0 Stdout:10055_10056_10056_0 Stdout:10056_10057_10057_0 Stdout:10057_10058_10058_0 Stdout:10058_10059_10059_0 Stdout:10059_10060_10060_0 Stdout:10060_10061_10061_0 Stdout:1006_1007_1007_0 Stdout:10061_10062_10062_0 Stdout:10062_10063_10063_0 Stdout:10063_10064_10064_0 Stdout:10064_10065_10065_0 Stdout:10065_10066_10066_0 Stdout:10066_10067_10067_0 Stdout:10067_10068_10068_0 Stdout:10068_10069_10069_0 Stdout:10069_10070_10070_0 Stdout:10070_10071_10071_0 Stdout:1007_1008_1008_0 Stdout:10071_10072_10072_0 Stdout:10072_10073_10073_0 Stdout:10073_10074_10074_0 Stdout:10074_10075_10075_0 Stdout:10075_10076_10076_0 Stdout:10076_10077_10077_0 Stdout:10077_10078_10078_0 Stdout:10078_10079_10079_0 Stdout:10079_10080_10080_0 Stdout:10080_10081_10081_0 Stdout:1008_1009_1009_0 Stdout:10081_10082_10082_0 Stdout:10082_10083_10083_0 Stdout:10083_10084_10084_0 Stdout:10084_10085_10085_0 Stdout:10085_10086_10086_0 Stdout:10086_10087_10087_0 Stdout:10087_10088_10088_0 Stdout:10088_10089_10089_0 Stdout:10089_10090_10090_0 Stdout:10090_10091_10091_0 Stdout:1009_1010_1010_0 Stdout:10091_10092_10092_0 Stdout:10092_10093_10093_0 Stdout:10093_10094_10094_0 Stdout:10094_10095_10095_0 Stdout:10095_10096_10096_0 Stdout:10096_10097_10097_0 Stdout:10097_10098_10098_0 Stdout:10098_10099_10099_0 Stdout:10099_10100_10100_0 Stdout:10100_10101_10101_0 Stdout:1010_1011_1011_0 Stdout:10101_10102_10102_0 Stdout:10102_10103_10103_0 Stdout:10103_10104_10104_0 Stdout:10104_10105_10105_0 Stdout:10105_10106_10106_0 Stdout:10106_10107_10107_0 Stdout:10107_10108_10108_0 Stdout:10108_10109_10109_0 Stdout:10109_10110_10110_0 Stdout:10110_10111_10111_0 Stdout:101_102_102_0 Stdout:1011_1012_1012_0 Stdout:10_11_11_0 Stdout:10111_10112_10112_0 Stdout:10112_10113_10113_0 Stdout:10113_10114_10114_0 Stdout:10114_10115_10115_0 Stdout:10115_10116_10116_0 Stdout:10116_10117_10117_0 Stdout:10117_10118_10118_0 Stdout:10118_10119_10119_0 Stdout:10119_10120_10120_0 Stdout:10120_10121_10121_0 Stdout:1012_1013_1013_0 Stdout:10121_10122_10122_0 Stdout:10122_10123_10123_0 Stdout:10123_10124_10124_0 Stdout:10124_10125_10125_0 Stdout:10125_10126_10126_0 Stdout:10126_10127_10127_0 Stdout:10127_10128_10128_0 Stdout:10128_10129_10129_0 Stdout:10129_10130_10130_0 Stdout:10130_10131_10131_0 Stdout:1013_1014_1014_0 Stdout:10131_10132_10132_0 Stdout:10132_10133_10133_0 Stdout:10133_10134_10134_0 Stdout:10134_10135_10135_0 Stdout:10135_10136_10136_0 Stdout:10136_10137_10137_0 Stdout:10137_10138_10138_0 Stdout:10138_10139_10139_0 Stdout:10139_10140_10140_0 Stdout:10140_10141_10141_0 Stdout:1014_1015_1015_0 Stdout:10141_10142_10142_0 Stdout:10142_10143_10143_0 Stdout:10143_10144_10144_0 Stdout:10144_10145_10145_0 Stdout:10145_10146_10146_0 Stdout:10146_10147_10147_0 Stdout:10147_10148_10148_0 Stdout:10148_10149_10149_0 Stdout:10149_10150_10150_0 Stdout:10150_10151_10151_0 Stdout:1015_1016_1016_0 Stdout:10151_10152_10152_0 Stdout:10152_10153_10153_0 Stdout:10153_10154_10154_0 Stdout:10154_10155_10155_0 Stdout:10155_10156_10156_0 Stdout:10156_10157_10157_0 Stdout:10157_10158_10158_0 Stdout:10158_10159_10159_0 Stdout:10159_10160_10160_0 Stdout:10160_10161_10161_0 Stdout:1016_1017_1017_0 Stdout:10161_10162_10162_0 Stdout:10162_10163_10163_0 Stdout:10163_10164_10164_0 Stdout:10164_10165_10165_0 Stdout:10165_10166_10166_0 Stdout:10166_10167_10167_0 Stdout:10167_10168_10168_0 Stdout:10168_10169_10169_0 Stdout:10169_10170_10170_0 Stdout:10170_10171_10171_0 Stdout:1017_1018_1018_0 Stdout:10171_10172_10172_0 Stdout:10172_10173_10173_0 Stdout:10173_10174_10174_0 Stdout:10174_10175_10175_0 Stdout:10175_10176_10176_0 Stdout:10176_10177_10177_0 Stdout:10177_10178_10178_0 Stdout:10178_10179_10179_0 Stdout:10179_10180_10180_0 Stdout:10180_10181_10181_0 Stdout:1018_1019_1019_0 Stdout:10181_10182_10182_0 Stdout:10182_10183_10183_0 Stdout:10183_10184_10184_0 Stdout:10184_10185_10185_0 Stdout:10185_10186_10186_0 Stdout:10186_10187_10187_0 Stdout:10187_10188_10188_0 Stdout:10188_10189_10189_0 Stdout:10189_10190_10190_0 Stdout:10190_10191_10191_0 Stdout:1019_1020_1020_0 Stdout:10191_10192_10192_0 Stdout:10192_10193_10193_0 Stdout:10193_10194_10194_0 Stdout:10194_10195_10195_0 Stdout:10195_10196_10196_0 Stdout:10196_10197_10197_0 Stdout:10197_10198_10198_0 Stdout:10198_10199_10199_0 Stdout:10199_10200_10200_0 Stdout:10200_10201_10201_0 Stdout:1020_1021_1021_0 Stdout:10201_10202_10202_0 Stdout:10202_10203_10203_0 Stdout:10203_10204_10204_0 Stdout:10204_10205_10205_0 Stdout:10205_10206_10206_0 Stdout:10206_10207_10207_0 Stdout:10207_10208_10208_0 Stdout:10208_10209_10209_0 Stdout:10209_10210_10210_0 Stdout:10210_10211_10211_0 Stdout:102_103_103_0 Stdout:1021_1022_1022_0 Stdout:10211_10212_10212_0 Stdout:10212_10213_10213_0 Stdout:10213_10214_10214_0 Stdout:10214_10215_10215_0 Stdout:10215_10216_10216_0 Stdout:10216_10217_10217_0 Stdout:10217_10218_10218_0 Stdout:10218_10219_10219_0 Stdout:10219_10220_10220_0 Stdout:10220_10221_10221_0 Stdout:1022_1023_1023_0 Stdout:10221_10222_10222_0 Stdout:10222_10223_10223_0 Stdout:10223_10224_10224_0 Stdout:10224_10225_10225_0 Stdout:10225_10226_10226_0 Stdout:10226_10227_10227_0 Stdout:10227_10228_10228_0 Stdout:10228_10229_10229_0 Stdout:10229_10230_10230_0 Stdout:10230_10231_10231_0 Stdout:1023_1024_1024_0 Stdout:10231_10232_10232_0 Stdout:10232_10233_10233_0 Stdout:10233_10234_10234_0 Stdout:10234_10235_10235_0 Stdout:10235_10236_10236_0 Stdout:10236_10237_10237_0 Stdout:10237_10238_10238_0 Stdout:10238_10239_10239_0 Stdout:10239_10240_10240_0 Stdout:10240_10241_10241_0 Stdout:1024_1025_1025_0 Stdout:10241_10242_10242_0 Stdout:10242_10243_10243_0 Stdout:10243_10244_10244_0 Stdout:10244_10245_10245_0 Stdout:10245_10246_10246_0 Stdout:10246_10247_10247_0 Stdout:10247_10248_10248_0 Stdout:10248_10249_10249_0 Stdout:10249_10250_10250_0 Stdout:10250_10251_10251_0 Stdout:1025_1026_1026_0 Stdout:10251_10252_10252_0 Stdout:10252_10253_10253_0 Stdout:10253_10254_10254_0 Stdout:10254_10255_10255_0 Stdout:10255_10256_10256_0 Stdout:10256_10257_10257_0 Stdout:10257_10258_10258_0 Stdout:10258_10259_10259_0 Stdout:10259_10260_10260_0 Stdout:10260_10261_10261_0 Stdout:1026_1027_1027_0 Stdout:10261_10262_10262_0 Stdout:10262_10263_10263_0 Stdout:10263_10264_10264_0 Stdout:10264_10265_10265_0 Stdout:10265_10266_10266_0 Stdout:10266_10267_10267_0 Stdout:10267_10268_10268_0 Stdout:10268_10269_10269_0 Stdout:10269_10270_10270_0 Stdout:10270_10271_10271_0 Stdout:1027_1028_1028_0 Stdout:10271_10272_10272_0 Stdout:10272_10273_10273_0 Stdout:10273_10274_10274_0 Stdout:10274_10275_10275_0 Stdout:10275_10276_10276_0 Stdout:10276_10277_10277_0 Stdout:10277_10278_10278_0 Stdout:10278_10279_10279_0 Stdout:10279_10280_10280_0 Stdout:10280_10281_10281_0 Stdout:1028_1029_1029_0 Stdout:10281_10282_10282_0 Stdout:10282_10283_10283_0 Stdout:10283_10284_10284_0 Stdout:10284_10285_10285_0 Stdout:10285_10286_10286_0 Stdout:10286_10287_10287_0 Stdout:10287_10288_10288_0 Stdout:10288_10289_10289_0 Stdout:10289_10290_10290_0 Stdout:10290_10291_10291_0 Stdout:1029_1030_1030_0 Stdout:10291_10292_10292_0 Stdout:10292_10293_10293_0 Stdout:10293_10294_10294_0 Stdout:10294_10295_10295_0 Stdout:10295_10296_10296_0 Stdout:10296_10297_10297_0 Stdout:10297_10298_10298_0 Stdout:10298_10299_10299_0 Stdout:10299_10300_10300_0 Stdout:10300_10301_10301_0 Stdout:1030_1031_1031_0 Stdout:10301_10302_10302_0 Stdout:10302_10303_10303_0 Stdout:10303_10304_10304_0 Stdout:10304_10305_10305_0 Stdout:10305_10306_10306_0 Stdout:10306_10307_10307_0 Stdout:10307_10308_10308_0 Stdout:10308_10309_10309_0 Stdout:10309_10310_10310_0 Stdout:10310_10311_10311_0 Stdout:103_104_104_0 Stdout:1031_1032_1032_0 Stdout:10311_10312_10312_0 Stdout:10312_10313_10313_0 Stdout:10313_10314_10314_0 Stdout:10314_10315_10315_0 Stdout:10315_10316_10316_0 Stdout:10316_10317_10317_0 Stdout:10317_10318_10318_0 Stdout:10318_10319_10319_0 Stdout:10319_10320_10320_0 Stdout:10320_10321_10321_0 Stdout:1032_1033_1033_0 Stdout:10321_10322_10322_0 Stdout:10322_10323_10323_0 Stdout:10323_10324_10324_0 Stdout:10324_10325_10325_0 Stdout:10325_10326_10326_0 Stdout:10326_10327_10327_0 Stdout:10327_10328_10328_0 Stdout:10328_10329_10329_0 Stdout:10329_10330_10330_0 Stdout:10330_10331_10331_0 Stdout:1033_1034_1034_0 Stdout:10331_10332_10332_0 Stdout:10332_10333_10333_0 Stdout:10333_10334_10334_0 Stdout:10334_10335_10335_0 Stdout:10335_10336_10336_0 Stdout:10336_10337_10337_0 Stdout:10337_10338_10338_0 Stdout:10338_10339_10339_0 Stdout:10339_10340_10340_0 Stdout:10340_10341_10341_0 Stdout:1034_1035_1035_0 Stdout:10341_10342_10342_0 Stdout:10342_10343_10343_0 Stdout:10343_10344_10344_0 Stdout:10344_10345_10345_0 Stdout:10345_10346_10346_0 Stdout:10346_10347_10347_0 Stdout:10347_10348_10348_0 Stdout:10348_10349_10349_0 Stdout:10349_10350_10350_0 Stdout:10350_10351_10351_0 Stdout:1035_1036_1036_0 Stdout:10351_10352_10352_0 Stdout:10352_10353_10353_0 Stdout:10353_10354_10354_0 Stdout:10354_10355_10355_0 Stdout:10355_10356_10356_0 Stdout:10356_10357_10357_0 Stdout:10357_10358_10358_0 Stdout:10358_10359_10359_0 Stdout:10359_10360_10360_0 Stdout:10360_10361_10361_0 Stdout:1036_1037_1037_0 Stdout:10361_10362_10362_0 Stdout:10362_10363_10363_0 Stdout:10363_10364_10364_0 Stdout:10364_10365_10365_0 Stdout:10365_10366_10366_0 Stdout:10366_10367_10367_0 Stdout:10367_10368_10368_0 Stdout:10368_10369_10369_0 Stdout:10369_10370_10370_0 Stdout:10370_10371_10371_0 Stdout:1037_1038_1038_0 Stdout:10371_10372_10372_0 Stdout:10372_10373_10373_0 Stdout:10373_10374_10374_0 Stdout:10374_10375_10375_0 Stdout:10375_10376_10376_0 Stdout:10376_10377_10377_0 Stdout:10377_10378_10378_0 Stdout:10378_10379_10379_0 Stdout:10379_10380_10380_0 Stdout:10380_10381_10381_0 Stdout:1038_1039_1039_0 Stdout:10381_10382_10382_0 Stdout:10382_10383_10383_0 Stdout:10383_10384_10384_0 Stdout:10384_10385_10385_0 Stdout:10385_10386_10386_0 Stdout:10386_10387_10387_0 Stdout:10387_10388_10388_0 Stdout:10388_10389_10389_0 Stdout:10389_10390_10390_0 Stdout:10390_10391_10391_0 Stdout:1039_1040_1040_0 Stdout:10391_10392_10392_0 Stdout:10392_10393_10393_0 Stdout:10393_10394_10394_0 Stdout:10394_10395_10395_0 Stdout:10395_10396_10396_0 Stdout:10396_10397_10397_0 Stdout:10397_10398_10398_0 Stdout:10398_10399_10399_0 Stdout:10399_10400_10400_0 Stdout:10400_10401_10401_0 Stdout:1040_1041_1041_0 Stdout:10401_10402_10402_0 Stdout:10402_10403_10403_0 Stdout:10403_10404_10404_0 Stdout:10404_10405_10405_0 Stdout:10405_10406_10406_0 Stdout:10406_10407_10407_0 Stdout:10407_10408_10408_0 Stdout:10408_10409_10409_0 Stdout:10409_10410_10410_0 Stdout:10410_10411_10411_0 Stdout:104_105_105_0 Stdout:1041_1042_1042_0 Stdout:10411_10412_10412_0 Stdout:10412_10413_10413_0 Stdout:10413_10414_10414_0 Stdout:10414_10415_10415_0 Stdout:10415_10416_10416_0 Stdout:10416_10417_10417_0 Stdout:10417_10418_10418_0 Stdout:10418_10419_10419_0 Stdout:10419_10420_10420_0 Stdout:10420_10421_10421_0 Stdout:1042_1043_1043_0 Stdout:10421_10422_10422_0 Stdout:10422_10423_10423_0 Stdout:10423_10424_10424_0 Stdout:10424_10425_10425_0 Stdout:10425_10426_10426_0 Stdout:10426_10427_10427_0 Stdout:10427_10428_10428_0 Stdout:10428_10429_10429_0 Stdout:10429_10430_10430_0 Stdout:10430_10431_10431_0 Stdout:1043_1044_1044_0 Stdout:10431_10432_10432_0 Stdout:10432_10433_10433_0 Stdout:10433_10434_10434_0 Stdout:10434_10435_10435_0 Stdout:10435_10436_10436_0 Stdout:10436_10437_10437_0 Stdout:10437_10438_10438_0 Stdout:10438_10439_10439_0 Stdout:10439_10440_10440_0 Stdout:10440_10441_10441_0 Stdout:1044_1045_1045_0 Stdout:10441_10442_10442_0 Stdout:10442_10443_10443_0 Stdout:10443_10444_10444_0 Stdout:10444_10445_10445_0 Stdout:10445_10446_10446_0 Stdout:10446_10447_10447_0 Stdout:10447_10448_10448_0 Stdout:10448_10449_10449_0 Stdout:10449_10450_10450_0 Stdout:10450_10451_10451_0 Stdout:1045_1046_1046_0 Stdout:10451_10452_10452_0 Stdout:10452_10453_10453_0 Stdout:10453_10454_10454_0 Stdout:10454_10455_10455_0 Stdout:10455_10456_10456_0 Stdout:10456_10457_10457_0 Stdout:10457_10458_10458_0 Stdout:10458_10459_10459_0 Stdout:10459_10460_10460_0 Stdout:10460_10461_10461_0 Stdout:1046_1047_1047_0 Stdout:10461_10462_10462_0 Stdout:10462_10463_10463_0 Stdout:10463_10464_10464_0 Stdout:10464_10465_10465_0 Stdout:10465_10466_10466_0 Stdout:10466_10467_10467_0 Stdout:10467_10468_10468_0 Stdout:10468_10469_10469_0 Stdout:10469_10470_10470_0 Stdout:10470_10471_10471_0 Stdout:1047_1048_1048_0 Stdout:10471_10472_10472_0 Stdout:10472_10473_10473_0 Stdout:10473_10474_10474_0 Stdout:10474_10475_10475_0 Stdout:10475_10476_10476_0 Stdout:10476_10477_10477_0 Stdout:10477_10478_10478_0 Stdout:10478_10479_10479_0 Stdout:10479_10480_10480_0 Stdout:10480_10481_10481_0 Stdout:1048_1049_1049_0 Stdout:10481_10482_10482_0 Stdout:10482_10483_10483_0 Stdout:10483_10484_10484_0 Stdout:10484_10485_10485_0 Stdout:10485_10486_10486_0 Stdout:10486_10487_10487_0 Stdout:10487_10488_10488_0 Stdout:10488_10489_10489_0 Stdout:10489_10490_10490_0 Stdout:10490_10491_10491_0 Stdout:1049_1050_1050_0 Stdout:10491_10492_10492_0 Stdout:10492_10493_10493_0 Stdout:10493_10494_10494_0 Stdout:10494_10495_10495_0 Stdout:10495_10496_10496_0 Stdout:10496_10497_10497_0 Stdout:10497_10498_10498_0 Stdout:10498_10499_10499_0 Stdout:10499_10500_10500_0 Stdout:10500_10501_10501_0 Stdout:1050_1051_1051_0 Stdout:10501_10502_10502_0 Stdout:10502_10503_10503_0 Stdout:10503_10504_10504_0 Stdout:10504_10505_10505_0 Stdout:10505_10506_10506_0 Stdout:10506_10507_10507_0 Stdout:10507_10508_10508_0 Stdout:10508_10509_10509_0 Stdout:10509_10510_10510_0 Stdout:10510_10511_10511_0 Stdout:105_106_106_0 Stdout:1051_1052_1052_0 Stdout:10511_10512_10512_0 Stdout:10512_10513_10513_0 Stdout:10513_10514_10514_0 Stdout:10514_10515_10515_0 Stdout:10515_10516_10516_0 Stdout:10516_10517_10517_0 Stdout:10517_10518_10518_0 Stdout:10518_10519_10519_0 Stdout:10519_10520_10520_0 Stdout:10520_10521_10521_0 Stdout:1052_1053_1053_0 Stdout:10521_10522_10522_0 Stdout:10522_10523_10523_0 Stdout:10523_10524_10524_0 Stdout:10524_10525_10525_0 Stdout:10525_10526_10526_0 Stdout:10526_10527_10527_0 Stdout:10527_10528_10528_0 Stdout:10528_10529_10529_0 Stdout:10529_10530_10530_0 Stdout:10530_10531_10531_0 Stdout:1053_1054_1054_0 Stdout:10531_10532_10532_0 Stdout:10532_10533_10533_0 Stdout:10533_10534_10534_0 Stdout:10534_10535_10535_0 Stdout:10535_10536_10536_0 Stdout:10536_10537_10537_0 Stdout:10537_10538_10538_0 Stdout:10538_10539_10539_0 Stdout:10539_10540_10540_0 Stdout:10540_10541_10541_0 Stdout:1054_1055_1055_0 Stdout:10541_10542_10542_0 Stdout:10542_10543_10543_0 Stdout:10543_10544_10544_0 Stdout:10544_10545_10545_0 Stdout:10545_10546_10546_0 Stdout:10546_10547_10547_0 Stdout:10547_10548_10548_0 Stdout:10548_10549_10549_0 Stdout:10549_10550_10550_0 Stdout:10550_10551_10551_0 Stdout:1055_1056_1056_0 Stdout:10551_10552_10552_0 Stdout:10552_10553_10553_0 Stdout:10553_10554_10554_0 Stdout:10554_10555_10555_0 Stdout:10555_10556_10556_0 Stdout:10556_10557_10557_0 Stdout:10557_10558_10558_0 Stdout:10558_10559_10559_0 Stdout:10559_10560_10560_0 Stdout:10560_10561_10561_0 Stdout:1056_1057_1057_0 Stdout:10561_10562_10562_0 Stdout:10562_10563_10563_0 Stdout:10563_10564_10564_0 Stdout:10564_10565_10565_0 Stdout:10565_10566_10566_0 Stdout:10566_10567_10567_0 Stdout:10567_10568_10568_0 Stdout:10568_10569_10569_0 Stdout:10569_10570_10570_0 Stdout:10570_10571_10571_0 Stdout:1057_1058_1058_0 Stdout:10571_10572_10572_0 Stdout:10572_10573_10573_0 Stdout:10573_10574_10574_0 Stdout:10574_10575_10575_0 Stdout:10575_10576_10576_0 Stdout:10576_10577_10577_0 Stdout:10577_10578_10578_0 Stdout:10578_10579_10579_0 Stdout:10579_10580_10580_0 Stdout:10580_10581_10581_0 Stdout:1058_1059_1059_0 Stdout:10581_10582_10582_0 Stdout:10582_10583_10583_0 Stdout:10583_10584_10584_0 Stdout:10584_10585_10585_0 Stdout:10585_10586_10586_0 Stdout:10586_10587_10587_0 Stdout:10587_10588_10588_0 Stdout:10588_10589_10589_0 Stdout:10589_10590_10590_0 Stdout:10590_10591_10591_0 Stdout:1059_1060_1060_0 Stdout:10591_10592_10592_0 Stdout:10592_10593_10593_0 Stdout:10593_10594_10594_0 Stdout:10594_10595_10595_0 Stdout:10595_10596_10596_0 Stdout:10596_10597_10597_0 Stdout:10597_10598_10598_0 Stdout:10598_10599_10599_0 Stdout:10599_10600_10600_0 Stdout:10600_10601_10601_0 Stdout:1060_1061_1061_0 Stdout:10601_10602_10602_0 Stdout:10602_10603_10603_0 Stdout:10603_10604_10604_0 Stdout:10604_10605_10605_0 Stdout:10605_10606_10606_0 Stdout:10606_10607_10607_0 Stdout:10607_10608_10608_0 Stdout:10608_10609_10609_0 Stdout:10609_10610_10610_0 Stdout:10610_10611_10611_0 Stdout:106_107_107_0 Stdout:1061_1062_1062_0 Stdout:10611_10612_10612_0 Stdout:10612_10613_10613_0 Stdout:10613_10614_10614_0 Stdout:10614_10615_10615_0 Stdout:10615_10616_10616_0 Stdout:10616_10617_10617_0 Stdout:10617_10618_10618_0 Stdout:10618_10619_10619_0 Stdout:10619_10620_10620_0 Stdout:10620_10621_10621_0 Stdout:1062_1063_1063_0 Stdout:10621_10622_10622_0 Stdout:10622_10623_10623_0 Stdout:10623_10624_10624_0 Stdout:10624_10625_10625_0 Stdout:10625_10626_10626_0 Stdout:10626_10627_10627_0 Stdout:10627_10628_10628_0 Stdout:10628_10629_10629_0 Stdout:10629_10630_10630_0 Stdout:10630_10631_10631_0 Stdout:1063_1064_1064_0 Stdout:10631_10632_10632_0 Stdout:10632_10633_10633_0 Stdout:10633_10634_10634_0 Stdout:10634_10635_10635_0 Stdout:10635_10636_10636_0 Stdout:10636_10637_10637_0 Stdout:10637_10638_10638_0 Stdout:10638_10639_10639_0 Stdout:10639_10640_10640_0 Stdout:10640_10641_10641_0 Stdout:1064_1065_1065_0 Stdout:10641_10642_10642_0 Stdout:10642_10643_10643_0 Stdout:10643_10644_10644_0 Stdout:10644_10645_10645_0 Stdout:10645_10646_10646_0 Stdout:10646_10647_10647_0 Stdout:10647_10648_10648_0 Stdout:10648_10649_10649_0 Stdout:10649_10650_10650_0 Stdout:10650_10651_10651_0 Stdout:1065_1066_1066_0 Stdout:10651_10652_10652_0 Stdout:10652_10653_10653_0 Stdout:10653_10654_10654_0 Stdout:10654_10655_10655_0 Stdout:10655_10656_10656_0 Stdout:10656_10657_10657_0 Stdout:10657_10658_10658_0 Stdout:10658_10659_10659_0 Stdout:10659_10660_10660_0 Stdout:10660_10661_10661_0 Stdout:1066_1067_1067_0 Stdout:10661_10662_10662_0 Stdout:10662_10663_10663_0 Stdout:10663_10664_10664_0 Stdout:10664_10665_10665_0 Stdout:10665_10666_10666_0 Stdout:10666_10667_10667_0 Stdout:10667_10668_10668_0 Stdout:10668_10669_10669_0 Stdout:10669_10670_10670_0 Stdout:10670_10671_10671_0 Stdout:1067_1068_1068_0 Stdout:10671_10672_10672_0 Stdout:10672_10673_10673_0 Stdout:10673_10674_10674_0 Stdout:10674_10675_10675_0 Stdout:10675_10676_10676_0 Stdout:10676_10677_10677_0 Stdout:10677_10678_10678_0 Stdout:10678_10679_10679_0 Stdout:10679_10680_10680_0 Stdout:10680_10681_10681_0 Stdout:1068_1069_1069_0 Stdout:10681_10682_10682_0 Stdout:10682_10683_10683_0 Stdout:10683_10684_10684_0 Stdout:10684_10685_10685_0 Stdout:10685_10686_10686_0 Stdout:10686_10687_10687_0 Stdout:10687_10688_10688_0 Stdout:10688_10689_10689_0 Stdout:10689_10690_10690_0 Stdout:10690_10691_10691_0 Stdout:1069_1070_1070_0 Stdout:10691_10692_10692_0 Stdout:10692_10693_10693_0 Stdout:10693_10694_10694_0 Stdout:10694_10695_10695_0 Stdout:10695_10696_10696_0 Stdout:10696_10697_10697_0 Stdout:10697_10698_10698_0 Stdout:10698_10699_10699_0 Stdout:10699_10700_10700_0 Stdout:10700_10701_10701_0 Stdout:1070_1071_1071_0 Stdout:10701_10702_10702_0 Stdout:10702_10703_10703_0 Stdout:10703_10704_10704_0 Stdout:10704_10705_10705_0 Stdout:10705_10706_10706_0 Stdout:10706_10707_10707_0 Stdout:10707_10708_10708_0 Stdout:10708_10709_10709_0 Stdout:10709_10710_10710_0 Stdout:10710_10711_10711_0 Stdout:107_108_108_0 Stdout:1071_1072_1072_0 Stdout:10711_10712_10712_0 Stdout:10712_10713_10713_0 Stdout:10713_10714_10714_0 Stdout:10714_10715_10715_0 Stdout:10715_10716_10716_0 Stdout:10716_10717_10717_0 Stdout:10717_10718_10718_0 Stdout:10718_10719_10719_0 Stdout:10719_10720_10720_0 Stdout:10720_10721_10721_0 Stdout:1072_1073_1073_0 Stdout:10721_10722_10722_0 Stdout:10722_10723_10723_0 Stdout:10723_10724_10724_0 Stdout:10724_10725_10725_0 Stdout:10725_10726_10726_0 Stdout:10726_10727_10727_0 Stdout:10727_10728_10728_0 Stdout:10728_10729_10729_0 Stdout:10729_10730_10730_0 Stdout:10730_10731_10731_0 Stdout:1073_1074_1074_0 Stdout:10731_10732_10732_0 Stdout:10732_10733_10733_0 Stdout:10733_10734_10734_0 Stdout:10734_10735_10735_0 Stdout:10735_10736_10736_0 Stdout:10736_10737_10737_0 Stdout:10737_10738_10738_0 Stdout:10738_10739_10739_0 Stdout:10739_10740_10740_0 Stdout:10740_10741_10741_0 Stdout:1074_1075_1075_0 Stdout:10741_10742_10742_0 Stdout:10742_10743_10743_0 Stdout:10743_10744_10744_0 Stdout:10744_10745_10745_0 Stdout:10745_10746_10746_0 Stdout:10746_10747_10747_0 Stdout:10747_10748_10748_0 Stdout:10748_10749_10749_0 Stdout:10749_10750_10750_0 Stdout:10750_10751_10751_0 Stdout:1075_1076_1076_0 Stdout:10751_10752_10752_0 Stdout:10752_10753_10753_0 Stdout:10753_10754_10754_0 Stdout:10754_10755_10755_0 Stdout:10755_10756_10756_0 Stdout:10756_10757_10757_0 Stdout:10757_10758_10758_0 Stdout:10758_10759_10759_0 Stdout:1_2_2_0 Executing query DROP TABLE IF EXISTS tbl SYNC on node [gw1] PASSED test_cancel_freeze/test.py::test_cancel_backup Executing query RESTORE TABLE system.functions ON CLUSTER 'cluster' FROM Disk('backups', '27') on node1 Executing query SELECT count(), sum(n) FROM other.merge on node Executing query INSERT INTO ordinary.mt1 (n) VALUES (24) on node Executing query SELECT number, linear_equation(number, 2, 1) FROM numbers(3) on node1 Executing query INSERT INTO ordinary.mt2 (n) VALUES (24) on node Executing query SELECT number, parity_str(number) FROM numbers(3) on node1 Executing query INSERT INTO ordinary.rmt1 (n) VALUES (24) on node Executing query SELECT number, linear_equation(number, 2, 1) FROM numbers(3) on node2 Executing query INSERT INTO ordinary.rmt2 (n) VALUES (24) on node Executing query SELECT number, parity_str(number) FROM numbers(3) on node2 Executing query INSERT INTO ordinary.mv1 (n) VALUES (24) on node Executing query SELECT name FROM system.functions WHERE name='parity_str' on node2 Executing query INSERT INTO ordinary.mv2 (n) VALUES (24) on node Executing query SELECT number, parity_str(number) FROM numbers(3) on node2 Executing query INSERT INTO ordinary.detached (n) VALUES (24) on node Executing query DROP FUNCTION linear_equation on node1 Executing query INSERT INTO other.mt1 (n) VALUES (15) on node Executing query DROP FUNCTION parity_str on node1 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_system_functions Executing query INSERT INTO other.mt2 (n) VALUES (15) on node Executing query INSERT INTO other.rmt1 (n) VALUES (15) on node Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query INSERT INTO other.rmt2 (n) VALUES (15) on node Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query INSERT INTO other.mv1 (n) VALUES (15) on node Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query INSERT INTO other.mv2 (n) VALUES (15) on node Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 Executing query INSERT INTO other.detached (n) VALUES (15) on node run container_id:roottestcancelfreeze-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm -r /var/lib/clickhouse/shadow/'] Command:[docker exec roottestcancelfreeze-gw1-node-1 bash -c rm -r /var/lib/clickhouse/shadow/] test_backup_restore_on_cluster/test.py::test_system_users Executing query CREATE USER u1 SETTINGS custom_a=123 on node1 Executing query INSERT INTO atomic.mt1 (n) VALUES (18) on node Command:[docker compose --env-file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/.env --project-name roottestcancelfreeze-gw1 --file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/docker-compose.yml stop --timeout 20] Executing query GRANT SELECT ON tbl TO u1 on node1 Executing query INSERT INTO atomic.mt2 (n) VALUES (18) on node Executing query CREATE USER u2 SETTINGS allow_backup=false on node1 Executing query INSERT INTO atomic.rmt1 (n) VALUES (18) on node Executing query GRANT CLUSTER ON *.* TO u2 on node1 Executing query INSERT INTO atomic.rmt2 (n) VALUES (18) on node Executing query BACKUP TABLE system.users ON CLUSTER 'cluster' TO Disk('backups', '28') on node1 Executing query INSERT INTO atomic.mv1 (n) VALUES (18) on node Executing query GRANT BACKUP ON system.users TO u2 on node1 Executing query INSERT INTO atomic.mv2 (n) VALUES (18) on node Executing query BACKUP TABLE system.users ON CLUSTER 'cluster' TO Disk('backups', '28') on node1 Executing query INSERT INTO atomic.detached (n) VALUES (18) on node Executing query DROP USER u1 on node1 Executing query SELECT count(), sum(n) FROM ordinary.merge on node Executing query RESTORE TABLE system.users ON CLUSTER 'cluster' FROM Disk('backups', '28') on node1 Executing query SELECT count(), sum(n) FROM ordinary.detached on node Executing query GRANT CREATE USER ON *.* TO u2 on node1 Executing query SELECT count(), sum(n) FROM other.merge on node Executing query RESTORE TABLE system.users ON CLUSTER 'cluster' FROM Disk('backups', '28') on node1 Executing query SELECT count(), sum(n) FROM other.detached on node Executing query GRANT SELECT ON tbl TO u2 WITH GRANT OPTION on node1 Executing query SELECT count(), sum(n) FROM atomic.merge on node Executing query RESTORE TABLE system.users ON CLUSTER 'cluster' FROM Disk('backups', '28') on node1 Executing query SELECT count(), sum(n) FROM atomic.detached on node Executing query SHOW CREATE USER u1 on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/.env --project-name roottestbackwardcompatibilityconvertordinary-gw9 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] [gw9] PASSED test_backward_compatibility/test_convert_ordinary.py::test_convert_ordinary_to_atomic Executing query SHOW GRANTS FOR u1 on node1 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_system_users Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Stderr: Container roottestcancelfreeze-gw1-node-1 Stopping Stderr: Container roottestcancelfreeze-gw1-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/.env --project-name roottestcancelfreeze-gw1 --file /ClickHouse/tests/integration/test_cancel_freeze/_instances-0-gw1/node/docker-compose.yml down --volumes] Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Stopping Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Stopped Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Stopping Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Stopping Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Stopping Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Stopped Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Stopped Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/.env --project-name roottestbackwardcompatibilityconvertordinary-gw9 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-convert_ordinary-0-gw9/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}') on node1 Stderr: Container roottestcancelfreeze-gw1-node-1 Stopping Stderr: Container roottestcancelfreeze-gw1-node-1 Stopped Stderr: Container roottestcancelfreeze-gw1-node-1 Removing Stderr: Container roottestcancelfreeze-gw1-node-1 Removed Stderr: Network roottestcancelfreeze-gw1_default Removing Stderr: Network roottestcancelfreeze-gw1_default Removed Cleanup called Docker networks for project roottestcancelfreeze-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcancelfreeze-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcancelfreeze-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcancelfreeze-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcancelfreeze-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Executing query CREATE TABLE mydb.tbl (x UInt8, y String) ENGINE=ReplicatedMergeTree ORDER BY tuple() on node1 Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Stopping Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Stopped Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Removing Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-node-1 Removed Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Stopping Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Stopping Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Stopping Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Stopped Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Removing Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Stopped Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Removing Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Stopped Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Removing Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo1-1 Removed Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo3-1 Removed Stderr: Container roottestbackwardcompatibilityconvertordinary-gw9-zoo2-1 Removed Stderr: Network roottestbackwardcompatibilityconvertordinary-gw9_default Removing Stderr: Network roottestbackwardcompatibilityconvertordinary-gw9_default Removed Cleanup called Docker networks for project roottestbackwardcompatibilityconvertordinary-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityconvertordinary-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityconvertordinary-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityconvertordinary-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityconvertordinary-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query ALTER TABLE mydb.tbl MODIFY COLUMN x String on node1 Stdout:1 Volumes pruned: 1 Running tests in /ClickHouse/tests/integration/test_cleanup_after_start/test.py test_cleanup_after_start/test.py::test_old_dirs_cleanup Cluster start called. is_up=False Docker networks for project roottestcleanupafterstart-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcleanupafterstart-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcleanupafterstart-gw9 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestcleanupafterstart-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcleanupafterstart-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcleanupafterstart-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcleanupafterstart-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcleanupafterstart-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/database Setup logs dir /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/.env --project-name roottestcleanupafterstart-gw9 --file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml pull] Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster' TO Disk('backups', '29') on node2 Executing query DROP DATABASE mydb ON CLUSTER 'cluster' SYNC on node1 Executing query RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM Disk('backups', '29') SETTINGS replica_num_in_backup=1 on node1 Executing query SELECT name, type FROM system.columns WHERE database='mydb' AND table='tbl' on node1 Executing query SELECT name, type FROM system.columns WHERE database='mydb' AND table='tbl' on node2 Executing query DROP DATABASE mydb ON CLUSTER 'cluster' SYNC on node1 Executing query RESTORE DATABASE mydb ON CLUSTER 'cluster' FROM Disk('backups', '29') SETTINGS replica_num_in_backup=2 on node2 Executing query SELECT name, type FROM system.columns WHERE database='mydb' AND table='tbl' on node1 Executing query SELECT name, type FROM system.columns WHERE database='mydb' AND table='tbl' on node2 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty Executing query CREATE DATABASE mydb ON CLUSTER 'cluster' ENGINE=Replicated('/clickhouse/path/','{shard}','{replica}') on node1 Executing query CREATE TABLE mydb.test (`x` UInt32) ENGINE = ReplicatedMergeTree ORDER BY x on node1 Executing query INSERT INTO mydb.test SELECT number AS x FROM numbers(10000000) on node1 Executing query BACKUP DATABASE mydb TO Disk('backups', '30') on node1 Executing query DROP DATABASE mydb SYNC on node1 Executing query RESTORE DATABASE mydb FROM Disk('backups', '30') on node1 Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 test_backup_restore_on_cluster/test.py::test_tables_dependency Executing query CREATE DATABASE mydb ON CLUSTER 'cluster3' on node1 Executing query CREATE TABLE mydb.src ON CLUSTER 'cluster' (x Int64, y String) ENGINE=MergeTree ORDER BY tuple() on node1 Executing query CREATE DICTIONARY mydb.dict ON CLUSTER 'cluster' (x Int64, y String) PRIMARY KEY x SOURCE(CLICKHOUSE(HOST 'localhost' PORT tcpPort() DB 'mydb' TABLE 'src')) LAYOUT(FLAT()) LIFETIME(0) on node1 Executing query CREATE TABLE mydb.dist1 (x Int64) ENGINE=Distributed('cluster', 'mydb', 'src') on node1 Executing query CREATE TABLE mydb.dist2 (x Int64) ENGINE=Distributed(cluster, 'mydb', 'src') on node3 Executing query CREATE TABLE mydb.clusterfunc1 AS cluster('cluster', 'mydb.src') on node1 Executing query CREATE TABLE mydb.clusterfunc2 AS cluster(cluster, mydb.src) on node1 Executing query CREATE TABLE mydb.clusterfunc3 AS cluster(cluster, 'mydb', 'src') on node1 Executing query CREATE TABLE mydb.clusterfunc4 AS cluster(cluster, dictionary(mydb.dict)) on node1 Executing query CREATE TABLE mydb.clusterfunc5 AS clusterAllReplicas(cluster, dictionary(mydb.dict)) on node1 Executing query CREATE TABLE mydb.clusterfunc6 AS cluster('cluster', 'mydb.src') on node3 Stderr: node1 Skipped - Image is already being pulled by zoo3 Stderr: zoo1 Skipped - Image is already being pulled by zoo3 Stderr: zoo2 Skipped - Image is already being pulled by zoo3 Stderr: zoo3 Pulling Stderr: zoo3 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper1/log', '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper1/config', '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper1/coordination', '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper2/log', '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper2/config', '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper2/coordination', '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper3/log', '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper3/config', '/ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/keeper3/coordination'] Command:[docker compose --project-name roottestcleanupafterstart-gw9 --env-file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Executing query CREATE TABLE mydb.clusterfunc7 AS cluster(cluster, mydb.src) on node3 Executing query CREATE TABLE mydb.clusterfunc8 AS cluster(cluster, 'mydb', 'src') on node3 Executing query CREATE TABLE mydb.clusterfunc9 AS cluster(cluster, dictionary(mydb.dict)) on node3 Stderr:time="2025-04-02T02:55:37Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestcleanupafterstart-gw9_default Creating Stderr: Network roottestcleanupafterstart-gw9_default Created Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Creating Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Creating Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Creating Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Created Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Created Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Created Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Starting Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Starting Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Starting Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Started Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Started Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Started Stderr:time="2025-04-02T02:55:38Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:55:38Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestcleanupafterstart-gw9-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query CREATE TABLE mydb.clusterfunc10 AS clusterAllReplicas(cluster, dictionary(mydb.dict)) on node3 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query BACKUP DATABASE mydb ON CLUSTER 'cluster3' TO Disk('backups', '31') on node3 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP DATABASE mydb on node3 Executing query RESTORE DATABASE mydb ON CLUSTER 'cluster3' FROM Disk('backups', '31') on node3 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster3' on node3 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.src has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.src has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.693881 [ 699 ] {11e9119d-e1c0-4d28-93d7-15c5d01de10e} RestorerFromBackup: Table mydb.src has no dependencies (level 0) run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.dict has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.dict has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.693925 [ 699 ] {11e9119d-e1c0-4d28-93d7-15c5d01de10e} RestorerFromBackup: Table mydb.dict has 1 dependencies: mydb.src (level 1) run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.dist1 has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.dist1 has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.693994 [ 699 ] {11e9119d-e1c0-4d28-93d7-15c5d01de10e} RestorerFromBackup: Table mydb.dist1 has 1 dependencies: mydb.src (level 1) run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc1 has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc1 has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.693966 [ 699 ] {11e9119d-e1c0-4d28-93d7-15c5d01de10e} RestorerFromBackup: Table mydb.clusterfunc1 has 1 dependencies: mydb.src (level 1) run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc2 has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc2 has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.694048 [ 699 ] {11e9119d-e1c0-4d28-93d7-15c5d01de10e} RestorerFromBackup: Table mydb.clusterfunc2 has 1 dependencies: mydb.src (level 1) run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc3 has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc3 has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.694022 [ 699 ] {11e9119d-e1c0-4d28-93d7-15c5d01de10e} RestorerFromBackup: Table mydb.clusterfunc3 has 1 dependencies: mydb.src (level 1) run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc4 has 1 dependencies: mydb.dict (level 2)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc4 has 1 dependencies: mydb.dict (level 2)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.694108 [ 699 ] {11e9119d-e1c0-4d28-93d7-15c5d01de10e} RestorerFromBackup: Table mydb.clusterfunc4 has 1 dependencies: mydb.dict (level 2) run container_id:roottestbackuprestoreoncluster-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc5 has 1 dependencies: mydb.dict (level 2)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc5 has 1 dependencies: mydb.dict (level 2)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.694081 [ 699 ] {11e9119d-e1c0-4d28-93d7-15c5d01de10e} RestorerFromBackup: Table mydb.clusterfunc5 has 1 dependencies: mydb.dict (level 2) run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.src has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.src has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.686718 [ 1584 ] {f5ae7515-5fd8-4d0f-8cdd-d84fa42a507b} RestorerFromBackup: Table mydb.src has no dependencies (level 0) run container_id:roottestbackuprestoreoncluster-gw2-node2-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.dict has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node2-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.dict has 1 dependencies: mydb.src (level 1)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.686765 [ 1584 ] {f5ae7515-5fd8-4d0f-8cdd-d84fa42a507b} RestorerFromBackup: Table mydb.dict has 1 dependencies: mydb.src (level 1) run container_id:roottestbackuprestoreoncluster-gw2-node3-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.dist2 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node3-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.dist2 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.688809 [ 672 ] {6dad2a0b-da1c-478b-94e9-9b8ecd4ff65c} RestorerFromBackup: Table mydb.dist2 has no dependencies (level 0) run container_id:roottestbackuprestoreoncluster-gw2-node3-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc6 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node3-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc6 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.688776 [ 672 ] {6dad2a0b-da1c-478b-94e9-9b8ecd4ff65c} RestorerFromBackup: Table mydb.clusterfunc6 has no dependencies (level 0) run container_id:roottestbackuprestoreoncluster-gw2-node3-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc7 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node3-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc7 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.688751 [ 672 ] {6dad2a0b-da1c-478b-94e9-9b8ecd4ff65c} RestorerFromBackup: Table mydb.clusterfunc7 has no dependencies (level 0) run container_id:roottestbackuprestoreoncluster-gw2-node3-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc8 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node3-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc8 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.688919 [ 672 ] {6dad2a0b-da1c-478b-94e9-9b8ecd4ff65c} RestorerFromBackup: Table mydb.clusterfunc8 has no dependencies (level 0) run container_id:roottestbackuprestoreoncluster-gw2-node3-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc9 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node3-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc9 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.688841 [ 672 ] {6dad2a0b-da1c-478b-94e9-9b8ecd4ff65c} RestorerFromBackup: Table mydb.clusterfunc9 has no dependencies (level 0) run container_id:roottestbackuprestoreoncluster-gw2-node3-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc10 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackuprestoreoncluster-gw2-node3-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "RestorerFromBackup: Table mydb.clusterfunc10 has no dependencies (level 0)" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:39.688697 [ 672 ] {6dad2a0b-da1c-478b-94e9-9b8ecd4ff65c} RestorerFromBackup: Table mydb.clusterfunc10 has no dependencies (level 0) Executing query DROP TABLE IF EXISTS tbl ON CLUSTER 'cluster3' SYNC on node1 [gw2] PASSED test_backup_restore_on_cluster/test.py::test_tables_dependency Executing query DROP TABLE IF EXISTS tbl2 ON CLUSTER 'cluster3' SYNC on node1 Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestcleanupafterstart-gw9-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestcleanupafterstart-gw9-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') Executing query DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster3' SYNC on node1 ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/.env --project-name roottestcleanupafterstart-gw9 --file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/.env --project-name roottestcleanupafterstart-gw9 --file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate] Executing query DROP DATABASE IF EXISTS mydb2 ON CLUSTER 'cluster3' SYNC on node1 Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Running Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Running Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Running Stderr: Container roottestcleanupafterstart-gw9-node1-1 Creating Stderr: Container roottestcleanupafterstart-gw9-node1-1 Created Stderr: Container roottestcleanupafterstart-gw9-node1-1 Starting Stderr: Container roottestcleanupafterstart-gw9-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestcleanupafterstart-gw9-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestcleanupafterstart-gw9-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestcleanupafterstart-gw9-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None Executing query DROP USER IF EXISTS u1, u2 ON CLUSTER 'cluster3' on node1 http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/.env --project-name roottestbackuprestoreoncluster-gw2 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/docker-compose.yml stop --timeout 20] http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/db533830febd0fb924da6e0a49dbe79fd9a7ac6743e19a2d00937b6f859119ad/json HTTP/1.1" 200 None ClickHouse node1 started Executing query CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_table', 'node1') PARTITION BY date ORDER BY id on node1 Executing query INSERT INTO test_table VALUES (toDate('2020-01-01'), 1, 10) on node1 Executing query SELECT count() FROM test_table on node1 run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestcleanupafterstart-gw9-node1-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 8 ? 00:00:01 clickhouse run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestcleanupafterstart-gw9-node1-1 bash -c pkill clickhouse] run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'mv /var/lib/clickhouse/data/default/test_table/20200101_0_0_0 /var/lib/clickhouse/data/default/test_table/delete_tmp_20200101_0_0_0'] Command:[docker exec --privileged roottestcleanupafterstart-gw9-node1-1 bash -c mv /var/lib/clickhouse/data/default/test_table/20200101_0_0_0 /var/lib/clickhouse/data/default/test_table/delete_tmp_20200101_0_0_0] run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestcleanupafterstart-gw9-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/6a87b187b6200d7ef83092794a4ee8ab3ee71b6cc1b47019a859ab58e30a9976/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/6a87b187b6200d7ef83092794a4ee8ab3ee71b6cc1b47019a859ab58e30a9976/json HTTP/1.1" 200 586 run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:786 Clickhouse process running. run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:786 Executing query select 20 on node1 Executing query select 20 on node1 run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'ls /var/lib/clickhouse/data/default/test_table/'] Command:[docker exec --privileged roottestcleanupafterstart-gw9-node1-1 bash -c ls /var/lib/clickhouse/data/default/test_table/] Stdout:20200101_0_0_0 Stdout:detached Stdout:format_version.txt Executing query SELECT count() FROM test_table on node1 run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "Removing temporary directory" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "Removing temporary directory" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:51.325033 [ 811 ] {} default.test_table (51ad82c1-1f8c-49a2-9279-e3f8546b3881): Removing temporary directory /var/lib/clickhouse/store/51a/51ad82c1-1f8c-49a2-9279-e3f8546b3881/delete_tmp_20200101_0_0_0/ run container_id:roottestcleanupafterstart-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "delete_tmp_20200101_0_0_0" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestcleanupafterstart-gw9-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "delete_tmp_20200101_0_0_0" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:55:51.325033 [ 811 ] {} default.test_table (51ad82c1-1f8c-49a2-9279-e3f8546b3881): Removing temporary directory /var/lib/clickhouse/store/51a/51ad82c1-1f8c-49a2-9279-e3f8546b3881/delete_tmp_20200101_0_0_0/ Command:[docker compose --env-file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/.env --project-name roottestcleanupafterstart-gw9 --file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] [gw9] PASSED test_cleanup_after_start/test.py::test_old_dirs_cleanup Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/.env --project-name roottestbackuprestoreoncluster-gw2 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-0-gw2/node3/docker-compose.yml down --volumes] Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Removing Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Removing Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Removing Stderr: Container roottestbackuprestoreoncluster-gw2-node2-1 Removed Stderr: Container roottestbackuprestoreoncluster-gw2-node3-1 Removed Stderr: Container roottestbackuprestoreoncluster-gw2-node1-1 Removed Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Stopping Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Removing Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Removing Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Stopped Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Removing Stderr: Container roottestbackuprestoreoncluster-gw2-zoo3-1 Removed Stderr: Container roottestbackuprestoreoncluster-gw2-zoo1-1 Removed Stderr: Container roottestbackuprestoreoncluster-gw2-zoo2-1 Removed Stderr: Network roottestbackuprestoreoncluster-gw2_default Removing Stderr: Network roottestbackuprestoreoncluster-gw2_default Removed Cleanup called Docker networks for project roottestbackuprestoreoncluster-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackuprestoreoncluster-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackuprestoreoncluster-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackuprestoreoncluster-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackuprestoreoncluster-gw2 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Stderr: Container roottestcleanupafterstart-gw9-node1-1 Stopping Stderr: Container roottestcleanupafterstart-gw9-node1-1 Stopped Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Stopping Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Stopping Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Stopping Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Stopped Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Stopped Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/.env --project-name roottestcleanupafterstart-gw9 --file /ClickHouse/tests/integration/test_cleanup_after_start/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] Stderr: Container roottestcleanupafterstart-gw9-node1-1 Stopping Stderr: Container roottestcleanupafterstart-gw9-node1-1 Stopped Stderr: Container roottestcleanupafterstart-gw9-node1-1 Removing Stderr: Container roottestcleanupafterstart-gw9-node1-1 Removed Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Stopping Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Stopping Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Stopping Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Stopped Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Removing Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Stopped Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Removing Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Stopped Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Removing Stderr: Container roottestcleanupafterstart-gw9-zoo1-1 Removed Stderr: Container roottestcleanupafterstart-gw9-zoo2-1 Removed Stderr: Container roottestcleanupafterstart-gw9-zoo3-1 Removed Stderr: Network roottestcleanupafterstart-gw9_default Removing Stderr: Network roottestcleanupafterstart-gw9_default Removed Cleanup called Docker networks for project roottestcleanupafterstart-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcleanupafterstart-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcleanupafterstart-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcleanupafterstart-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcleanupafterstart-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_compression_nested_columns/test.py::test_nested_compression_codec Running tests in /ClickHouse/tests/integration/test_compression_nested_columns/test.py Cluster start called. is_up=False Docker networks for project roottestcompressionnestedcolumns-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcompressionnestedcolumns-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcompressionnestedcolumns-gw9 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestcompressionnestedcolumns-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcompressionnestedcolumns-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcompressionnestedcolumns-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcompressionnestedcolumns-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestcompressionnestedcolumns-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/database Setup logs dir /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/database Setup logs dir /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/.env --project-name roottestcompressionnestedcolumns-gw9 --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/docker-compose.yml pull] Stderr: zoo1 Skipped - Image is already being pulled by node1 Stderr: zoo2 Skipped - Image is already being pulled by node1 Stderr: zoo3 Skipped - Image is already being pulled by node1 Stderr: node2 Skipped - Image is already being pulled by node1 Stderr: node1 Pulling Stderr: node1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper1/log', '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper1/config', '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper1/coordination', '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper2/log', '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper2/config', '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper2/coordination', '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper3/log', '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper3/config', '/ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/keeper3/coordination'] Command:[docker compose --project-name roottestcompressionnestedcolumns-gw9 --env-file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2025-04-02T02:56:04Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestcompressionnestedcolumns-gw9_default Creating Stderr: Network roottestcompressionnestedcolumns-gw9_default Created Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Creating Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Creating Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Creating Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Created Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Created Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Created Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Starting Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Starting Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Starting Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Started Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Started Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Started Stderr:time="2025-04-02T02:56:05Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:56:05Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/.env --project-name roottestcompressionnestedcolumns-gw9 --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/.env --project-name roottestcompressionnestedcolumns-gw9 --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/docker-compose.yml up -d --no-recreate] Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Running Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Running Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Running Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Creating Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Creating Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Created Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Created Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Starting Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Starting Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Started Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.6... http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/fefad8439f1163ad3b7e719dea7a17704a1466a2a8539c0f3ceda7b56bf55bff/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestcompressionnestedcolumns-gw9-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/3d336373de071a95fb2fc88696460f3dd2a807a7d87d9c56ae287bc4a6c1ce73/json HTTP/1.1" 200 None ClickHouse node2 started Executing query CREATE TABLE compression_table ( key UInt64, column_ok Nullable(UInt64) CODEC(Delta, LZ4), column_array Array(Array(UInt64)) CODEC(T64, LZ4), column_bad LowCardinality(Int64) CODEC(Delta) ) ENGINE = ReplicatedMergeTree('/t', '0') ORDER BY tuple() PARTITION BY key SETTINGS min_rows_for_wide_part = 0, min_bytes_for_wide_part = 0, replace_long_file_name_to_hash = 0; on node1 Executing query CREATE TABLE compression_table ( key UInt64, column_ok Nullable(UInt64) CODEC(Delta, LZ4), column_array Array(Array(UInt64)) CODEC(T64, LZ4), column_bad LowCardinality(Int64) CODEC(Delta) ) ENGINE = ReplicatedMergeTree('/t', '1') ORDER BY tuple() PARTITION BY key SETTINGS min_rows_for_wide_part = 0, min_bytes_for_wide_part = 0, replace_long_file_name_to_hash = 0; on node2 Executing query INSERT INTO compression_table VALUES (1, 1, [[77]], 32) on node1 Executing query SYSTEM SYNC REPLICA compression_table on node2 Executing query DETACH TABLE compression_table on node1 Executing query DETACH TABLE compression_table on node2 Executing query ATTACH TABLE compression_table on node1 Executing query ATTACH TABLE compression_table on node2 run container_id:roottestcompressionnestedcolumns-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_ok.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node1-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_ok.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0091 run container_id:roottestcompressionnestedcolumns-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_ok.null.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node1-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_ok.null.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0082 run container_id:roottestcompressionnestedcolumns-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node1-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0091 run container_id:roottestcompressionnestedcolumns-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.size0.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node2-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.size0.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0082 run container_id:roottestcompressionnestedcolumns-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.size1.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node2-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.size1.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0082 run container_id:roottestcompressionnestedcolumns-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_bad.dict.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node2-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_bad.dict.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0092 run container_id:roottestcompressionnestedcolumns-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_bad.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node1-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_bad.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0002 run container_id:roottestcompressionnestedcolumns-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_ok.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node2-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_ok.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0091 run container_id:roottestcompressionnestedcolumns-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_ok.null.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node2-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_ok.null.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0082 run container_id:roottestcompressionnestedcolumns-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node1-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0091 run container_id:roottestcompressionnestedcolumns-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.size0.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node2-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.size0.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0082 run container_id:roottestcompressionnestedcolumns-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.size1.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node2-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_array.size1.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0082 run container_id:roottestcompressionnestedcolumns-gw9-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_bad.dict.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node2-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_bad.dict.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0092 run container_id:roottestcompressionnestedcolumns-gw9-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_bad.bin | od -x -N 1 | head -n 1 | awk '{print $2}'"] Command:[docker exec roottestcompressionnestedcolumns-gw9-node1-1 bash -c tail -c +17 /var/lib/clickhouse/data/default/compression_table/1_0_0_0/column_bad.bin | od -x -N 1 | head -n 1 | awk '{print $2}'] Stdout:0002 Command:[docker compose --env-file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/.env --project-name roottestcompressionnestedcolumns-gw9 --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/docker-compose.yml stop --timeout 20] [gw9] PASSED test_compression_nested_columns/test.py::test_nested_compression_codec Stderr: node1 Skipped - Image is already being pulled by node2 Stderr: zoo1 Skipped - Image is already being pulled by node2 Stderr: zoo2 Skipped - Image is already being pulled by node2 Stderr: zoo3 Skipped - Image is already being pulled by node2 Stderr: azurite1 Pulling Stderr: node2 Pulling Stderr: node2 Pulled Stderr: f18232174bc9 Pulling fs layer Stderr: cb2bde55f71f Pulling fs layer Stderr: 9d0e0719fbe0 Pulling fs layer Stderr: 6f063dbd7a5d Pulling fs layer Stderr: f9e3e3d8f042 Pulling fs layer Stderr: a39ef2f62dc8 Pulling fs layer Stderr: 9a21c6b23f0e Pulling fs layer Stderr: efeb7b313b67 Pulling fs layer Stderr: 6fef65209747 Pulling fs layer Stderr: 3d377e512a83 Pulling fs layer Stderr: a39ef2f62dc8 Waiting Stderr: 9a21c6b23f0e Waiting Stderr: efeb7b313b67 Waiting Stderr: 6fef65209747 Waiting Stderr: 3d377e512a83 Waiting Stderr: 6f063dbd7a5d Waiting Stderr: f9e3e3d8f042 Waiting Stderr: 9d0e0719fbe0 Downloading [> ] 15.58kB/1.261MB Stderr: f18232174bc9 Downloading [> ] 48.34kB/3.642MB Stderr: cb2bde55f71f Downloading [> ] 506.1kB/50.34MB Stderr: 9d0e0719fbe0 Downloading [===========> ] 293.1kB/1.261MB Stderr: f18232174bc9 Downloading [========> ] 588.1kB/3.642MB Stderr: 9d0e0719fbe0 Verifying Checksum Stderr: 9d0e0719fbe0 Download complete Stderr: cb2bde55f71f Downloading [==> ] 2.03MB/50.34MB Stderr: f18232174bc9 Downloading [===============================> ] 2.308MB/3.642MB Stderr: f18232174bc9 Verifying Checksum Stderr: f18232174bc9 Download complete Stderr: f18232174bc9 Extracting [> ] 65.54kB/3.642MB Stderr: cb2bde55f71f Downloading [=======> ] 7.109MB/50.34MB Stderr: f18232174bc9 Extracting [==================================================>] 3.642MB/3.642MB Stderr: f18232174bc9 Extracting [==================================================>] 3.642MB/3.642MB Stderr: f18232174bc9 Pull complete Stderr: cb2bde55f71f Downloading [=================> ] 17.27MB/50.34MB Stderr: cb2bde55f71f Downloading [=============================> ] 29.97MB/50.34MB Stderr: cb2bde55f71f Downloading [========================================> ] 40.63MB/50.34MB Stderr: cb2bde55f71f Verifying Checksum Stderr: cb2bde55f71f Download complete Stderr: cb2bde55f71f Extracting [> ] 524.3kB/50.34MB Stderr: cb2bde55f71f Extracting [======> ] 6.816MB/50.34MB Stderr: cb2bde55f71f Extracting [===============> ] 15.73MB/50.34MB Stderr: cb2bde55f71f Extracting [======================> ] 23.07MB/50.34MB Stderr: cb2bde55f71f Extracting [===============================> ] 31.46MB/50.34MB Stderr: cb2bde55f71f Extracting [========================================> ] 40.89MB/50.34MB Stderr: cb2bde55f71f Extracting [==============================================> ] 46.66MB/50.34MB Stderr: cb2bde55f71f Extracting [===============================================> ] 47.71MB/50.34MB Stderr: cb2bde55f71f Extracting [===============================================> ] 48.23MB/50.34MB Stderr: cb2bde55f71f Extracting [================================================> ] 48.76MB/50.34MB Stderr: cb2bde55f71f Extracting [=================================================> ] 49.81MB/50.34MB Stderr: cb2bde55f71f Extracting [=================================================> ] 50.33MB/50.34MB Stderr: cb2bde55f71f Extracting [==================================================>] 50.34MB/50.34MB Stderr: cb2bde55f71f Pull complete Stderr: 9d0e0719fbe0 Extracting [=> ] 32.77kB/1.261MB Stderr: 9d0e0719fbe0 Extracting [==================================================>] 1.261MB/1.261MB Stderr: 9d0e0719fbe0 Extracting [==================================================>] 1.261MB/1.261MB Stderr: 9d0e0719fbe0 Pull complete Stderr: f9e3e3d8f042 Downloading [==================================================>] 116B/116B Stderr: f9e3e3d8f042 Verifying Checksum Stderr: f9e3e3d8f042 Download complete Stderr: 6f063dbd7a5d Downloading [==================================================>] 446B/446B Stderr: 6f063dbd7a5d Verifying Checksum Stderr: 6f063dbd7a5d Download complete Stderr: 6f063dbd7a5d Extracting [==================================================>] 446B/446B Stderr: 6f063dbd7a5d Extracting [==================================================>] 446B/446B Stderr: a39ef2f62dc8 Downloading [> ] 3.29kB/209.4kB Stderr: 6f063dbd7a5d Pull complete Stderr: f9e3e3d8f042 Extracting [==================================================>] 116B/116B Stderr: f9e3e3d8f042 Extracting [==================================================>] 116B/116B Stderr: f9e3e3d8f042 Pull complete Stderr: a39ef2f62dc8 Downloading [==================================================>] 209.4kB/209.4kB Stderr: a39ef2f62dc8 Download complete Stderr: a39ef2f62dc8 Extracting [=======> ] 32.77kB/209.4kB Stderr: a39ef2f62dc8 Extracting [==================================================>] 209.4kB/209.4kB Stderr: a39ef2f62dc8 Extracting [==================================================>] 209.4kB/209.4kB Stderr: a39ef2f62dc8 Pull complete Stderr: efeb7b313b67 Downloading [=> ] 15.58kB/458.8kB Stderr: 9a21c6b23f0e Downloading [> ] 15.58kB/794kB Stderr: efeb7b313b67 Downloading [===================================> ] 325.9kB/458.8kB Stderr: efeb7b313b67 Verifying Checksum Stderr: efeb7b313b67 Download complete Stderr: 9a21c6b23f0e Downloading [=======================> ] 375.1kB/794kB Stderr: 6fef65209747 Downloading [> ] 375.1kB/36.34MB Stderr: 9a21c6b23f0e Download complete Stderr: 9a21c6b23f0e Extracting [==> ] 32.77kB/794kB Stderr: 6fef65209747 Downloading [==> ] 1.506MB/36.34MB Stderr: 9a21c6b23f0e Extracting [==============================> ] 491.5kB/794kB Stderr: 6fef65209747 Downloading [======> ] 4.52MB/36.34MB Stderr: 9a21c6b23f0e Extracting [===============================================> ] 753.7kB/794kB Stderr: 9a21c6b23f0e Extracting [==================================================>] 794kB/794kB Stderr: 9a21c6b23f0e Pull complete Stderr: efeb7b313b67 Extracting [===> ] 32.77kB/458.8kB Stderr: 6fef65209747 Downloading [==============> ] 10.55MB/36.34MB Stderr: efeb7b313b67 Extracting [==================================================>] 458.8kB/458.8kB Stderr: efeb7b313b67 Extracting [==================================================>] 458.8kB/458.8kB Stderr: efeb7b313b67 Pull complete Stderr: 6fef65209747 Downloading [==============================> ] 22.23MB/36.34MB Stderr: 6fef65209747 Downloading [======================================> ] 28.26MB/36.34MB Stderr: 6fef65209747 Verifying Checksum Stderr: 6fef65209747 Download complete Stderr: 6fef65209747 Extracting [> ] 393.2kB/36.34MB Stderr: 6fef65209747 Extracting [=> ] 1.18MB/36.34MB Stderr: 6fef65209747 Extracting [==> ] 1.573MB/36.34MB Stderr: 6fef65209747 Extracting [===> ] 2.359MB/36.34MB Stderr: 6fef65209747 Extracting [====> ] 3.146MB/36.34MB Stderr: 6fef65209747 Extracting [=====> ] 3.932MB/36.34MB Stderr: 6fef65209747 Extracting [======> ] 4.719MB/36.34MB Stderr: 6fef65209747 Extracting [=======> ] 5.505MB/36.34MB Stderr: 6fef65209747 Extracting [========> ] 5.898MB/36.34MB Stderr: 6fef65209747 Extracting [========> ] 6.291MB/36.34MB Stderr: 6fef65209747 Extracting [=========> ] 7.078MB/36.34MB Stderr: 6fef65209747 Extracting [==========> ] 7.864MB/36.34MB Stderr: 6fef65209747 Extracting [===========> ] 8.258MB/36.34MB Stderr: 6fef65209747 Extracting [===========> ] 8.651MB/36.34MB Stderr: 6fef65209747 Extracting [============> ] 9.437MB/36.34MB Stderr: 6fef65209747 Extracting [=============> ] 9.83MB/36.34MB Stderr: 6fef65209747 Extracting [=================> ] 12.98MB/36.34MB Stderr: 6fef65209747 Extracting [==================> ] 13.37MB/36.34MB Stderr: 6fef65209747 Extracting [===================> ] 14.16MB/36.34MB Stderr: 6fef65209747 Extracting [====================> ] 14.94MB/36.34MB Stderr: 6fef65209747 Extracting [=====================> ] 15.34MB/36.34MB Stderr: 6fef65209747 Extracting [======================> ] 16.12MB/36.34MB Stderr: 6fef65209747 Extracting [=======================> ] 17.3MB/36.34MB Stderr: 6fef65209747 Extracting [============================> ] 20.45MB/36.34MB Stderr: 6fef65209747 Extracting [=====================================> ] 27.53MB/36.34MB Stderr: 6fef65209747 Extracting [=========================================> ] 29.88MB/36.34MB Stderr: 6fef65209747 Extracting [=============================================> ] 33.03MB/36.34MB Stderr: 6fef65209747 Extracting [==================================================>] 36.34MB/36.34MB Stderr: 6fef65209747 Pull complete Stderr: 3d377e512a83 Downloading [==================================================>] 2.862kB/2.862kB Stderr: 3d377e512a83 Verifying Checksum Stderr: 3d377e512a83 Download complete Stderr: 3d377e512a83 Extracting [==================================================>] 2.862kB/2.862kB Stderr: 3d377e512a83 Extracting [==================================================>] 2.862kB/2.862kB Stderr: 3d377e512a83 Pull complete Stderr: azurite1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper1/log', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper1/config', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper1/coordination', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper2/log', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper2/config', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper2/coordination', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper3/log', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper3/config', '/ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/keeper3/coordination'] Command:[docker compose --project-name roottestazureblobstoragezerocopyreplication-gw0 --env-file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2025-04-02T02:56:16Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestazureblobstoragezerocopyreplication-gw0_default Creating Stderr: Network roottestazureblobstoragezerocopyreplication-gw0_default Created Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Creating Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Creating Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Creating Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Created Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Created Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Created Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Starting Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Starting Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Starting Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Started Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Started Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Started Stderr:time="2025-04-02T02:56:17Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:56:17Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.2.2, port:2181, use_ssl:False Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/.env --project-name roottestcompressionnestedcolumns-gw9 --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_compression_nested_columns/_instances-0-gw9/node2/docker-compose.yml down --volumes] Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Removing Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Removing Stderr: Container roottestcompressionnestedcolumns-gw9-node1-1 Removed Stderr: Container roottestcompressionnestedcolumns-gw9-node2-1 Removed Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Stopping Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Removing Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Removing Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Stopped Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Removing Stderr: Container roottestcompressionnestedcolumns-gw9-zoo3-1 Removed Stderr: Container roottestcompressionnestedcolumns-gw9-zoo1-1 Removed Stderr: Container roottestcompressionnestedcolumns-gw9-zoo2-1 Removed Stderr: Network roottestcompressionnestedcolumns-gw9_default Removing Stderr: Network roottestcompressionnestedcolumns-gw9_default Removed Cleanup called Docker networks for project roottestcompressionnestedcolumns-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestcompressionnestedcolumns-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestcompressionnestedcolumns-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestcompressionnestedcolumns-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Unstopped containers: {} No running containers for project: roottestcompressionnestedcolumns-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.2.4, port:2181, use_ssl:False Connecting to 172.16.2.4(172.16.2.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.2.3, port:2181, use_ssl:False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') Trying to create Azurite instance by command docker compose --project-name roottestazureblobstoragezerocopyreplication-gw0 --env-file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --verbose up -d Command:[docker compose --project-name roottestazureblobstoragezerocopyreplication-gw0 --env-file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --verbose up -d] Stderr:time="2025-04-02T02:56:24Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Volume "roottestazureblobstoragezerocopyreplication-gw0_data1-1" Creating Stderr: Volume "roottestazureblobstoragezerocopyreplication-gw0_data1-1" Created Stderr:time="2025-04-02T02:56:24Z" level=warning msg="Found orphan containers ([roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 roottestazureblobstoragezerocopyreplication-gw0-zoo2-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Creating Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Created Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Starting Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Started Stderr:time="2025-04-02T02:56:24Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:56:24Z" level=debug msg="otel error" error="" Trying to connect to Azurite Request URL: 'http://127.0.0.1:30000/devstoreaccount1/?restype=REDACTED&comp=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '10ea6706-0f6e-11f0-bd81-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request Starting new HTTP connection (1): 127.0.0.1:30000 http://127.0.0.1:30000 "GET /devstoreaccount1/?restype=account&comp=properties HTTP/1.1" 200 0 Response status: 200 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-client-request-id': '10ea6706-0f6e-11f0-bd81-0242ac110002' 'x-ms-request-id': '8ca0fe3c-a22e-4bbf-a718-0d99675d69db' 'x-ms-version': 'REDACTED' 'date': 'Wed, 02 Apr 2025 02:56:25 GMT' 'x-ms-sku-name': 'REDACTED' 'x-ms-account-kind': 'REDACTED' 'x-ms-is-hns-enabled': 'REDACTED' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' {'client_request_id': '10ea6706-0f6e-11f0-bd81-0242ac110002', 'request_id': '8ca0fe3c-a22e-4bbf-a718-0d99675d69db', 'version': '2025-05-05', 'date': datetime.datetime(2025, 4, 2, 2, 56, 25, tzinfo=datetime.timezone.utc), 'sku_name': 'Standard_RAGRS', 'account_kind': 'StorageV2', 'is_hns_enabled': False} Request URL: 'http://127.0.0.1:30000/devstoreaccount1/?comp=REDACTED&prefix=REDACTED&include=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '10f0b7dc-0f6e-11f0-bd81-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request http://127.0.0.1:30000 "GET /devstoreaccount1/?comp=list&prefix=azurite-container&include= HTTP/1.1" 200 None Response status: 200 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-client-request-id': '10f0b7dc-0f6e-11f0-bd81-0242ac110002' 'x-ms-request-id': 'ff4b37ec-ae3b-471f-9406-4520bd8508cb' 'x-ms-version': 'REDACTED' 'content-type': 'application/xml' 'Date': 'Wed, 02 Apr 2025 02:56:25 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Transfer-Encoding': 'chunked' Request URL: 'http://127.0.0.1:30000/devstoreaccount1/azurite-container?restype=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '10f2697e-0f6e-11f0-bd81-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request http://127.0.0.1:30000 "GET /devstoreaccount1/azurite-container?restype=container HTTP/1.1" 404 None Response status: 404 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'x-ms-error-code': 'ContainerNotFound' 'x-ms-request-id': '7a0d155b-03b8-45aa-83eb-8ac364ed4db6' 'content-type': 'application/xml' 'Date': 'Wed, 02 Apr 2025 02:56:25 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Transfer-Encoding': 'chunked' azurite container 'azurite-container' doesn't exist, creating it Request URL: 'http://127.0.0.1:30000/devstoreaccount1/azurite-container?restype=REDACTED' Request method: 'PUT' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '10f39a06-0f6e-11f0-bd81-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request http://127.0.0.1:30000 "PUT /devstoreaccount1/azurite-container?restype=container HTTP/1.1" 201 0 Response status: 201 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'etag': '"0x2062C12D8A6B340"' 'last-modified': 'Wed, 02 Apr 2025 02:56:25 GMT' 'x-ms-client-request-id': '10f39a06-0f6e-11f0-bd81-0242ac110002' 'x-ms-request-id': '09e4c89a-a226-4127-9307-c93e0e41b4a6' 'x-ms-version': 'REDACTED' 'Date': 'Wed, 02 Apr 2025 02:56:25 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Content-Length': '0' ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env --project-name roottestazureblobstoragezerocopyreplication-gw0 --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env --project-name roottestazureblobstoragezerocopyreplication-gw0 --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/docker-compose.yml up -d --no-recreate] Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Running Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Running Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Running Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Running Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Creating Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Creating Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Created Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Created Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Starting Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Starting Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Started Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.2.7... http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/37132de3fbc2a738ed227a592ef006551bead983f98770e42738d4df942ee7c4/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.2.6... http://localhost:None "GET /v1.46/containers/roottestazureblobstoragezerocopyreplication-gw0-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8a5f501f2f232941d4b82812cfb4dbee85540000169faaa722490286a937a648/json HTTP/1.1" 200 None ClickHouse node2 started Cluster started Executing query CREATE TABLE blob_storage_table ON CLUSTER test_cluster ( id Int64, data String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/blob_storage_table', '{replica}') ORDER BY id SETTINGS storage_policy='blob_storage_policy',old_parts_lifetime=1 on node1 Executing query SELECT COUNT(*) FROM blob_storage_table FORMAT Values on node1 Executing query INSERT INTO blob_storage_table VALUES (0,'data'),(1,'data') on node1 Executing query SYSTEM SYNC REPLICA blob_storage_table on node2 Executing query SELECT * FROM blob_storage_table order by id FORMAT Values on node1 Executing query SELECT * FROM blob_storage_table order by id FORMAT Values on node2 Request URL: 'http://127.0.0.1:30000/devstoreaccount1/cont?restype=REDACTED&comp=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '13040646-0f6e-11f0-bd81-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request http://127.0.0.1:30000 "GET /devstoreaccount1/cont?restype=container&comp=list HTTP/1.1" 200 None Response status: 200 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'content-type': 'application/xml' 'x-ms-client-request-id': '13040646-0f6e-11f0-bd81-0242ac110002' 'x-ms-request-id': 'aaa811d2-6b1b-4d10-b624-1f330cee3d9b' 'x-ms-version': 'REDACTED' 'date': 'Wed, 02 Apr 2025 02:56:28 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Transfer-Encoding': 'chunked' Executing query INSERT INTO blob_storage_table VALUES (2,'data'),(3,'data') on node2 Executing query SYSTEM SYNC REPLICA blob_storage_table on node1 Executing query SELECT * FROM blob_storage_table order by id FORMAT Values on node2 Executing query SELECT * FROM blob_storage_table order by id FORMAT Values on node1 Request URL: 'http://127.0.0.1:30000/devstoreaccount1/cont?restype=REDACTED&comp=REDACTED' Request method: 'GET' Request headers: 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.10.12 (Linux-5.15.0-130-generic-x86_64-with-glibc2.35)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '13998cf2-0f6e-11f0-bd81-0242ac110002' 'Authorization': 'REDACTED' No body was attached to the request http://127.0.0.1:30000 "GET /devstoreaccount1/cont?restype=container&comp=list HTTP/1.1" 200 None Response status: 200 Response headers: 'Server': 'Azurite-Blob/3.34.0' 'content-type': 'application/xml' 'x-ms-client-request-id': '13998cf2-0f6e-11f0-bd81-0242ac110002' 'x-ms-request-id': '322ad9ef-d25d-4137-83f7-0dbc0ff86af7' 'x-ms-version': 'REDACTED' 'date': 'Wed, 02 Apr 2025 02:56:29 GMT' 'Connection': 'keep-alive' 'Keep-Alive': 'REDACTED' 'Transfer-Encoding': 'chunked' Executing query DROP TABLE blob_storage_table ON CLUSTER test_cluster SYNC on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env --project-name roottestazureblobstoragezerocopyreplication-gw0 --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/docker-compose.yml stop --timeout 20] [gw0] PASSED test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/.env --project-name roottestazureblobstoragezerocopyreplication-gw0 --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_azurite.yml --file /ClickHouse/tests/integration/test_azure_blob_storage_zero_copy_replication/_instances-0-gw0/node2/docker-compose.yml down --volumes] Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Removing Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Removing Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node1-1 Removed Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-node2-1 Removed Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Stopping Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Removing Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Removing Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Removing Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Stopped Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Removing Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo1-1 Removed Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-azurite1-1 Removed Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo3-1 Removed Stderr: Container roottestazureblobstoragezerocopyreplication-gw0-zoo2-1 Removed Stderr: Volume roottestazureblobstoragezerocopyreplication-gw0_data1-1 Removing Stderr: Network roottestazureblobstoragezerocopyreplication-gw0_default Removing Stderr: Volume roottestazureblobstoragezerocopyreplication-gw0_data1-1 Removed Stderr: Network roottestazureblobstoragezerocopyreplication-gw0_default Removed Cleanup called Docker networks for project roottestazureblobstoragezerocopyreplication-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestazureblobstoragezerocopyreplication-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestazureblobstoragezerocopyreplication-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestazureblobstoragezerocopyreplication-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestazureblobstoragezerocopyreplication-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column Running tests in /ClickHouse/tests/integration/test_backward_compatibility/test_normalized_count_comparison.py Cluster start called. is_up=False Docker networks for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilitynormalizedcountcomparison-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilitynormalizedcountcomparison-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/.env --project-name roottestbackwardcompatibilitynormalizedcountcomparison-gw0 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/docker-compose.yml pull] Stderr: node2 Pulling Stderr: node1 Pulling Stderr: node1 Pulled Stderr: node2 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/.env --project-name roottestbackwardcompatibilitynormalizedcountcomparison-gw0 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/.env --project-name roottestbackwardcompatibilitynormalizedcountcomparison-gw0 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/docker-compose.yml up -d --no-recreate] Stderr: Network roottestbackwardcompatibilitynormalizedcountcomparison-gw0_default Creating Stderr: Network roottestbackwardcompatibilitynormalizedcountcomparison-gw0_default Created Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Creating Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Creating Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Created Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Created Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Starting Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Starting Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Started Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.2... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/10ffa2f32117d23d027cc0da345e8f298a4772b564722ea6c7fb0166feeee447/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.1.3... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/2ef02914494931c431db2174eae2eb6e86a2398dec2e8ba1d4714f65882129c2/json HTTP/1.1" 200 None ClickHouse node2 started Executing query create table tab (x UInt64, y String, z Nullable(Int64)) engine = Memory on node1 Executing query create table tab (x UInt64, y String, z Nullable(Int64)) engine = Memory on node2 Executing query insert into tab values (1, 'a', null) on node1 Executing query insert into tab values (1, 'a', null) on node2 Executing query select count(), count(1), count(x), count(y), count(z) from remote('node{1,2}', default, tab) on node1 Executing query select count(), count(1), count(x), count(y), count(z) from remote('node{1,2}', default, tab) on node2 Executing query drop table tab on node1 Executing query drop table tab on node2 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/.env --project-name roottestbackwardcompatibilitynormalizedcountcomparison-gw0 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/docker-compose.yml stop --timeout 20] [gw0] PASSED test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Stopping Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Stopping Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Stopped Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/.env --project-name roottestbackwardcompatibilitynormalizedcountcomparison-gw0 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-normalized_count_comparison-0-gw0/node2/docker-compose.yml down --volumes] Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Stopping Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Stopping Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Stopped Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Removing Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Stopped Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Removing Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node2-1 Removed Stderr: Container roottestbackwardcompatibilitynormalizedcountcomparison-gw0-node1-1 Removed Stderr: Network roottestbackwardcompatibilitynormalizedcountcomparison-gw0_default Removing Stderr: Network roottestbackwardcompatibilitynormalizedcountcomparison-gw0_default Removed Cleanup called Docker networks for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilitynormalizedcountcomparison-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilitynormalizedcountcomparison-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilitynormalizedcountcomparison-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Executing query SYSTEM SYNC REPLICA replicated_mt_1 on node1 Executing query CHECK TABLE replicated_mt_1 PARTITION 201901 on node1 Executing query SELECT count() from replicated_mt_1 on node1 Executing query SELECT path FROM system.parts WHERE table = 'replicated_mt_1' and name = '201901_0_0_0' on node2 run container_id:roottestchecktable-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm -r /var/lib/clickhouse/store/370/37029df9-8590-40d7-a946-19cf4d67a445/201901_0_0_0//*'] Command:[docker exec --privileged roottestchecktable-gw3-node2-1 bash -c rm -r /var/lib/clickhouse/store/370/37029df9-8590-40d7-a946-19cf4d67a445/201901_0_0_0//*] Executing query CHECK TABLE replicated_mt_1 PARTITION 201901 on node2 Executing query SYSTEM SYNC REPLICA replicated_mt_1 on node1 Executing query CHECK TABLE replicated_mt_1 PARTITION 201901 on node1 Executing query SELECT count() from replicated_mt_1 on node1 [gw3] PASSED test_check_table/test.py::test_check_replicated_table_corruption[-_0-.bin] test_check_table/test.py::test_check_replicated_table_simple[-_0] Executing query DROP TABLE IF EXISTS replicated_mt SYNC on node1 Executing query CREATE TABLE replicated_mt(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/replicated_mt__0', 'node1') PARTITION BY toYYYYMM(date) ORDER BY id on node1 Executing query DROP TABLE IF EXISTS replicated_mt SYNC on node2 Executing query CREATE TABLE replicated_mt(date Date, id UInt32, value Int32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/replicated_mt__0', 'node2') PARTITION BY toYYYYMM(date) ORDER BY id on node2 Executing query INSERT INTO replicated_mt VALUES (toDate('2019-02-01'), 1, 10), (toDate('2019-02-01'), 2, 12) on node1 Executing query SYSTEM SYNC REPLICA replicated_mt on node2 Executing query SELECT count() from replicated_mt on node1 Executing query SELECT count() from replicated_mt on node2 Executing query CHECK TABLE replicated_mt on node1 Executing query CHECK TABLE replicated_mt on node2 Executing query INSERT INTO replicated_mt VALUES (toDate('2019-01-02'), 3, 10), (toDate('2019-01-02'), 4, 12) on node2 Executing query SYSTEM SYNC REPLICA replicated_mt on node1 Executing query SELECT count() from replicated_mt on node1 Executing query SELECT count() from replicated_mt on node2 Executing query CHECK TABLE replicated_mt PARTITION 201901 on node1 Executing query CHECK TABLE replicated_mt PARTITION 201901 on node2 Executing query CHECK TABLE replicated_mt on node2 Executing query CHECK TABLE replicated_mt PART '201801_0_0_0' on node2 Executing query CHECK TABLE replicated_mt PART '201902_0_0_0' on node2 Command:[docker compose --env-file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/.env --project-name roottestchecktable-gw3 --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/docker-compose.yml stop --timeout 20] [gw3] PASSED test_check_table/test.py::test_check_replicated_table_simple[-_0] Stderr: Container roottestchecktable-gw3-node2-1 Stopping Stderr: Container roottestchecktable-gw3-node1-1 Stopping Stderr: Container roottestchecktable-gw3-node1-1 Stopped Stderr: Container roottestchecktable-gw3-node2-1 Stopped Stderr: Container roottestchecktable-gw3-zoo1-1 Stopping Stderr: Container roottestchecktable-gw3-zoo3-1 Stopping Stderr: Container roottestchecktable-gw3-zoo2-1 Stopping Stderr: Container roottestchecktable-gw3-zoo3-1 Stopped Stderr: Container roottestchecktable-gw3-zoo2-1 Stopped Stderr: Container roottestchecktable-gw3-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/.env --project-name roottestchecktable-gw3 --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_check_table/_instances-0-gw3/node2/docker-compose.yml down --volumes] Stderr: Container roottestchecktable-gw3-node1-1 Stopping Stderr: Container roottestchecktable-gw3-node2-1 Stopping Stderr: Container roottestchecktable-gw3-node1-1 Stopped Stderr: Container roottestchecktable-gw3-node1-1 Removing Stderr: Container roottestchecktable-gw3-node2-1 Stopped Stderr: Container roottestchecktable-gw3-node2-1 Removing Stderr: Container roottestchecktable-gw3-node2-1 Removed Stderr: Container roottestchecktable-gw3-node1-1 Removed Stderr: Container roottestchecktable-gw3-zoo3-1 Stopping Stderr: Container roottestchecktable-gw3-zoo1-1 Stopping Stderr: Container roottestchecktable-gw3-zoo2-1 Stopping Stderr: Container roottestchecktable-gw3-zoo3-1 Stopped Stderr: Container roottestchecktable-gw3-zoo3-1 Removing Stderr: Container roottestchecktable-gw3-zoo1-1 Stopped Stderr: Container roottestchecktable-gw3-zoo1-1 Removing Stderr: Container roottestchecktable-gw3-zoo2-1 Stopped Stderr: Container roottestchecktable-gw3-zoo2-1 Removing Stderr: Container roottestchecktable-gw3-zoo1-1 Removed Stderr: Container roottestchecktable-gw3-zoo3-1 Removed Stderr: Container roottestchecktable-gw3-zoo2-1 Removed Stderr: Network roottestchecktable-gw3_default Removing Stderr: Network roottestchecktable-gw3_default Removed Cleanup called Docker networks for project roottestchecktable-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestchecktable-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestchecktable-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestchecktable-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestchecktable-gw3 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts Running tests in /ClickHouse/tests/integration/test_backward_compatibility/test_vertical_merges_from_compact_parts.py Cluster start called. is_up=False Docker networks for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_backward_compatibility/configs/no_compress_marks.xml', '/ClickHouse/tests/integration/test_backward_compatibility/configs/no_allow_vertical_merges_from_compact_to_wide_parts.xml'] to /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/database Setup logs dir /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/.env --project-name roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/docker-compose.yml pull] Stderr: node2 Skipped - Image is already being pulled by zoo2 Stderr: zoo3 Skipped - Image is already being pulled by zoo2 Stderr: zoo1 Skipped - Image is already being pulled by zoo2 Stderr: node1 Pulling Stderr: zoo2 Pulling Stderr: zoo2 Pulled Stderr: node1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper1/log', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper1/config', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper1/coordination', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper2/log', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper2/config', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper2/coordination', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper3/log', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper3/config', '/ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/keeper3/coordination'] Command:[docker compose --project-name roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2025-04-02T02:58:41Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3_default Creating Stderr: Network roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3_default Created Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Creating Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Creating Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Creating Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Created Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Created Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Created Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Starting Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Starting Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Starting Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Started Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Started Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Started Stderr:time="2025-04-02T02:58:41Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T02:58:41Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/.env --project-name roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/.env --project-name roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/docker-compose.yml up -d --no-recreate] Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Running Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Running Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Running Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Creating Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Creating Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Created Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Created Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Starting Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Starting Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Started Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.6... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4523aba854bf51b65ced3accb7f390db0b34b484b0e029c697721058e6c120dd/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4523aba854bf51b65ced3accb7f390db0b34b484b0e029c697721058e6c120dd/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4523aba854bf51b65ced3accb7f390db0b34b484b0e029c697721058e6c120dd/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4523aba854bf51b65ced3accb7f390db0b34b484b0e029c697721058e6c120dd/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4523aba854bf51b65ced3accb7f390db0b34b484b0e029c697721058e6c120dd/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ba465d4ae543551863f97a465cd222cb208bde0db208299525825ad5452b578b/json HTTP/1.1" 200 None ClickHouse node2 started Executing query CREATE TABLE t_vertical_merges (id UInt64, v1 UInt64, v2 UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/0/t_vertical_merges', '0') ORDER BY id SETTINGS index_granularity = 50, vertical_merge_algorithm_min_rows_to_activate = 1, vertical_merge_algorithm_min_columns_to_activate = 1, min_bytes_for_wide_part = 0, min_rows_for_wide_part = 100 on node1 Executing query CREATE TABLE t_vertical_merges (id UInt64, v1 UInt64, v2 UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/0/t_vertical_merges', '1') ORDER BY id SETTINGS index_granularity = 50, vertical_merge_algorithm_min_rows_to_activate = 1, vertical_merge_algorithm_min_columns_to_activate = 1, min_bytes_for_wide_part = 0, min_rows_for_wide_part = 100 on node2 Executing query INSERT INTO t_vertical_merges SELECT number, number, number FROM numbers(60) on node2 Executing query INSERT INTO t_vertical_merges SELECT number * 2, number, number FROM numbers(60) on node2 Executing query OPTIMIZE TABLE t_vertical_merges FINAL on node2 Executing query SYSTEM SYNC REPLICA t_vertical_merges on node1 Executing query SYSTEM FLUSH LOGS on node2 Executing query SYSTEM FLUSH LOGS on node1 Executing query SELECT merge_algorithm, part_type FROM system.part_log WHERE event_type = 'MergeParts' AND table = 't_vertical_merges' AND part_name = 'all_0_1_1'; on node2 Executing query SELECT merge_algorithm, part_type FROM system.part_log WHERE event_type = 'MergeParts' AND table = 't_vertical_merges' AND part_name = 'all_0_1_1'; on node1 Executing query ALTER TABLE t_vertical_merges MODIFY SETTING allow_vertical_merges_from_compact_to_wide_parts = 1 on node2 Executing query INSERT INTO t_vertical_merges SELECT number * 3, number, number FROM numbers(60) on node2 Executing query OPTIMIZE TABLE t_vertical_merges FINAL on node2 Executing query SYSTEM SYNC REPLICA t_vertical_merges on node1 Executing query SYSTEM FLUSH LOGS on node2 Executing query SYSTEM FLUSH LOGS on node1 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "CHECKSUM_DOESNT_MATCH" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "CHECKSUM_DOESNT_MATCH" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "CHECKSUM_DOESNT_MATCH" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "CHECKSUM_DOESNT_MATCH" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:58:49.518109 [ 583 ] {ec545db0-ce56-49d6-91a3-ac228dd0aacf::all_0_1_1} ec545db0-ce56-49d6-91a3-ac228dd0aacf::all_0_1_1 (MergeFromLogEntryTask): Code: 40. DB::Exception: Different number of files: 3 compressed (expected 3) and 6 uncompressed ones (expected 5). (CHECKSUM_DOESNT_MATCH) (version 25.2.2.173926.altinityantalya.173926 (official build)). Data after merge is not byte-identical to data on another replicas. There could be several reasons: 1. Using newer version of compression library after server update. 2. Using another compression method. 3. Non-deterministic compression algorithm (highly unlikely). 4. Non-deterministic merge algorithm due to logical error in code. 5. Data corruption in memory due to bug in code. 6. Data corruption in memory due to hardware issue. 7. Manual modification of source data after server startup. 8. Manual modification of checksums stored in ZooKeeper. 9. Part format related settings like 'enable_mixed_granularity_parts' are different on different replicas. We will download merged part from replica to force byte-identical result. List of files in local parts: Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:58:51.837086 [ 596 ] {ec545db0-ce56-49d6-91a3-ac228dd0aacf::all_0_2_2} ec545db0-ce56-49d6-91a3-ac228dd0aacf::all_0_2_2 (MergeFromLogEntryTask): Code: 40. DB::Exception: Different number of files: 3 compressed (expected 3) and 6 uncompressed ones (expected 5). (CHECKSUM_DOESNT_MATCH) (version 25.2.2.173926.altinityantalya.173926 (official build)). Data after merge is not byte-identical to data on another replicas. There could be several reasons: 1. Using newer version of compression library after server update. 2. Using another compression method. 3. Non-deterministic compression algorithm (highly unlikely). 4. Non-deterministic merge algorithm due to logical error in code. 5. Data corruption in memory due to bug in code. 6. Data corruption in memory due to hardware issue. 7. Manual modification of source data after server startup. 8. Manual modification of checksums stored in ZooKeeper. 9. Part format related settings like 'enable_mixed_granularity_parts' are different on different replicas. We will download merged part from replica to force byte-identical result. List of files in local parts: Executing query SELECT merge_algorithm, part_type FROM system.part_log WHERE event_type = 'MergeParts' AND table = 't_vertical_merges' AND part_name = 'all_0_2_2'; on node2 Executing query SELECT merge_algorithm, part_type FROM system.part_log WHERE event_type = 'MergeParts' AND table = 't_vertical_merges' AND part_name = 'all_0_2_2'; on node1 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -15 clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c pkill -15 clickhouse] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse server' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'cp /usr/bin/clickhouse /usr/share/clickhouse_original'] Command:[docker exec -u root roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c cp /usr/bin/clickhouse /usr/share/clickhouse_original] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'cp /usr/share/clickhouse_fresh /usr/bin/clickhouse && chmod 777 /usr/bin/clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c cp /usr/share/clickhouse_fresh /usr/bin/clickhouse && chmod 777 /usr/bin/clickhouse] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "echo 'restart_with_latest_version: From version' && /usr/share/clickhouse_original server --version && echo 'To version' /usr/share/clickhouse_fresh server --version"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c echo 'restart_with_latest_version: From version' && /usr/share/clickhouse_original server --version && echo 'To version' /usr/share/clickhouse_fresh server --version] Stdout:restart_with_latest_version: From version Stdout:ClickHouse server version 23.3.22.3 (official build). Stdout:To version /usr/share/clickhouse_fresh server --version run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- '] Command:[docker exec -u 0 roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:496 Executing query select 20 on node1 Executing query select 20 on node1 Executing query select 20 on node1 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 8 ? 00:00:05 clickhouse run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c pkill clickhouse] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/04ba0c037fa611669f7ea457d85b48fe1ee816472b8733711bccdd8e74c79574/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/04ba0c037fa611669f7ea457d85b48fe1ee816472b8733711bccdd8e74c79574/json HTTP/1.1" 200 587 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:804 Clickhouse process running. run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:804 Executing query select 20 on node2 Executing query select 20 on node2 Executing query ALTER TABLE t_vertical_merges MODIFY SETTING allow_vertical_merges_from_compact_to_wide_parts = 1 on node1 Executing query INSERT INTO t_vertical_merges SELECT number * 4, number, number FROM numbers(60) on node2 Executing query OPTIMIZE TABLE t_vertical_merges FINAL on node2 Executing query SYSTEM SYNC REPLICA t_vertical_merges on node1 Executing query SYSTEM FLUSH LOGS on node2 Executing query SYSTEM FLUSH LOGS on node1 run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "CHECKSUM_DOESNT_MATCH" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "CHECKSUM_DOESNT_MATCH" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "CHECKSUM_DOESNT_MATCH" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "CHECKSUM_DOESNT_MATCH" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:59:13.569742 [ 1409 ] {ec545db0-ce56-49d6-91a3-ac228dd0aacf::all_0_3_3} ec545db0-ce56-49d6-91a3-ac228dd0aacf::all_0_3_3 (MergeFromLogEntryTask): Code: 40. DB::Exception: Different number of files: 3 compressed (expected 7) and 6 uncompressed ones (expected 2). (CHECKSUM_DOESNT_MATCH) (version 25.2.2.173926.altinityantalya.173926 (official build)). Data after merge is not byte-identical to data on another replicas. There could be several reasons: 1. Using newer version of compression library after server update. 2. Using another compression method. 3. Non-deterministic compression algorithm (highly unlikely). 4. Non-deterministic merge algorithm due to logical error in code. 5. Data corruption in memory due to bug in code. 6. Data corruption in memory due to hardware issue. 7. Manual modification of source data after server startup. 8. Manual modification of checksums stored in ZooKeeper. 9. Part format related settings like 'enable_mixed_granularity_parts' are different on different replicas. We will download merged part from replica to force byte-identical result. List of files in local parts: run container_id:roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "Different number of files" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "Different number of files" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 02:59:13.569742 [ 1409 ] {ec545db0-ce56-49d6-91a3-ac228dd0aacf::all_0_3_3} ec545db0-ce56-49d6-91a3-ac228dd0aacf::all_0_3_3 (MergeFromLogEntryTask): Code: 40. DB::Exception: Different number of files: 3 compressed (expected 7) and 6 uncompressed ones (expected 2). (CHECKSUM_DOESNT_MATCH) (version 25.2.2.173926.altinityantalya.173926 (official build)). Data after merge is not byte-identical to data on another replicas. There could be several reasons: 1. Using newer version of compression library after server update. 2. Using another compression method. 3. Non-deterministic compression algorithm (highly unlikely). 4. Non-deterministic merge algorithm due to logical error in code. 5. Data corruption in memory due to bug in code. 6. Data corruption in memory due to hardware issue. 7. Manual modification of source data after server startup. 8. Manual modification of checksums stored in ZooKeeper. 9. Part format related settings like 'enable_mixed_granularity_parts' are different on different replicas. We will download merged part from replica to force byte-identical result. List of files in local parts: Executing query SELECT merge_algorithm, part_type FROM system.part_log WHERE event_type = 'MergeParts' AND table = 't_vertical_merges' AND part_name = 'all_0_3_3'; on node2 Executing query SELECT merge_algorithm, part_type FROM system.part_log WHERE event_type = 'MergeParts' AND table = 't_vertical_merges' AND part_name = 'all_0_3_3'; on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/.env --project-name roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/docker-compose.yml stop --timeout 20] [gw3] PASSED test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/.env --project-name roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backward_compatibility/_instances-vertical_merges_from_compact_parts-0-gw3/node2/docker-compose.yml down --volumes] Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Removing Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Removing Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node2-1 Removed Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-node1-1 Removed Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Stopping Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Removing Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Removing Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Stopped Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Removing Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo1-1 Removed Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo3-1 Removed Stderr: Container roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-zoo2-1 Removed Stderr: Network roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3_default Removing Stderr: Network roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3_default Removed Cleanup called Docker networks for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestbackwardcompatibilityverticalmergesfromcompactparts-gw3 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 ============================== slowest durations =============================== 311.88s call test_check_table/test.py::test_check_replicated_table_corruption[-_0-.bin] 177.43s setup test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication 114.81s call test_backward_compatibility/test_convert_ordinary.py::test_convert_ordinary_to_atomic 58.74s call test_cancel_freeze/test.py::test_cancel_backup 42.59s call test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 40.53s call test_backward_compatibility/test_functions.py::test_aggregate_states 28.64s call test_cluster_all_replicas/test.py::test_skip_unavailable_replica[two_shards_three_nodes] 27.34s call test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 25.11s setup test_cluster_all_replicas/test.py::test_cluster 20.72s setup test_check_table/test.py::test_check_all_tables 20.56s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 20.22s setup test_cluster_discovery/test_password.py::test_connect_with_password 19.87s teardown test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication 19.51s setup test_compatibility_merge_tree_settings/test.py::test_check_projections_compatibility 18.96s setup test_config_substitutions/test.py::test_allow_databases 18.80s setup test_backward_compatibility/test_functions.py::test_aggregate_states 18.72s setup test_broken_part_during_merge/test.py::test_merge_and_part_corruption 18.50s setup test_allowed_client_hosts/test.py::test_allowed_host 18.39s setup test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 18.00s setup test_cleanup_after_start/test.py::test_old_dirs_cleanup 16.94s teardown test_alternative_keeper_config/test.py::test_create_insert 16.76s setup test_compression_nested_columns/test.py::test_nested_compression_codec 16.27s setup test_async_connect_to_multiple_ips/test.py::test 16.00s call test_backup_restore_on_cluster/test.py::test_shutdown_waits_for_backup 15.85s teardown test_config_substitutions/test.py::test_include_config 15.30s setup test_backup_restore/test.py::test_attach_partition 14.80s setup test_async_metrics_in_cgroup/test.py::test_normalized_user_cpu 14.73s setup test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default 14.54s setup test_access_for_functions/test.py::test_access_rights_for_function 14.37s teardown test_cluster_all_replicas/test.py::test_skip_unavailable_replica[two_shards_three_nodes] 13.86s setup test_buffer_profile/test.py::test_buffer_profile 13.75s setup test_authentication/test.py::test_authentication_fail 13.38s setup test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 13.10s setup test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 12.97s setup test_backward_compatibility/test_cte_distributed.py::test_cte_distributed 12.83s setup test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column 12.73s setup test_alter_codec/test.py::test_alter_codec_index 12.56s setup test_alternative_keeper_config/test.py::test_create_insert 12.51s setup test_build_sets_from_multiple_threads/test.py::test_set 12.23s call test_check_table/test.py::test_check_normal_table_corruption[] 12.18s setup test_alter_on_mixed_type_cluster/test.py::test_alter_on_cluter_non_replicated 11.41s teardown test_backup_restore_on_cluster/test.py::test_tables_dependency 10.51s setup test_cancel_freeze/test.py::test_cancel_backup 10.25s setup test_concurrent_queries_for_all_users_restriction/test.py::test_exception_message 10.05s setup test_backward_compatibility/test_convert_ordinary.py::test_convert_ordinary_to_atomic 9.07s setup test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Atomic] 9.07s teardown test_cancel_freeze/test.py::test_cancel_backup 8.41s setup test_backward_compatibility/test_select_aggregate_alias_column.py::test_select_aggregate_alias_column 8.28s call test_check_table/test.py::test_check_all_tables 8.26s teardown test_check_table/test.py::test_check_replicated_table_simple[-_0] 8.05s call test_backup_restore/test.py::test_attach_partition 7.57s teardown test_alter_on_mixed_type_cluster/test.py::test_alter_replicated_on_cluster 7.50s call test_cgroup_limit/test.py::test_cgroup_cpu_limit 7.38s call test_config_substitutions/test.py::test_config_multiple_zk_substitutions 7.36s call test_cleanup_after_start/test.py::test_old_dirs_cleanup 7.19s call test_backup_restore_on_cluster/test.py::test_tables_dependency 6.95s call test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[one_shard_three_nodes] 6.94s call test_backup_restore/test.py::test_replace_partition 6.84s call test_backup_restore_on_cluster/test.py::test_replicated_database_async 6.81s call test_access_for_functions/test.py::test_ignore_obsolete_grant_on_database 6.64s call test_backup_restore/test.py::test_restore 6.60s teardown test_compression_nested_columns/test.py::test_nested_compression_codec 6.56s call test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[two_shards_three_nodes] 6.35s call test_alter_on_mixed_type_cluster/test.py::test_alter_on_cluter_non_replicated 6.29s setup test_attach_table_normalizer/test.py::test_attach_substr 6.25s call test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_limit_reached 6.23s call test_cluster_all_replicas/test.py::test_skip_unavailable_replica[one_shard_three_nodes] 6.20s teardown test_async_connect_to_multiple_ips/test.py::test 6.17s teardown test_aggregation_memory_efficient/test.py::test_remote 5.80s teardown test_compatibility_merge_tree_settings/test.py::test_config_overrides_compatibility 5.80s call test_alter_on_mixed_type_cluster/test.py::test_alter_replicated_on_cluster 5.62s call test_backup_restore_on_cluster/test.py::test_required_privileges 5.55s call test_async_connect_to_multiple_ips/test.py::test 5.50s call test_check_table/test.py::test_check_replicated_table_simple[-_0] 5.47s call test_backup_restore_on_cluster/test.py::test_replicated_database 5.47s call test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Atomic] 5.41s call test_backup_restore_on_cluster/test.py::test_mutation 5.37s setup test_aggregation_memory_efficient/test.py::test_remote 5.36s teardown test_backup_restore/test.py::test_restore 5.30s call test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 5.24s teardown test_custom_settings/test.py::test_custom_settings 5.21s teardown test_authentication/test.py::test_authentication_pass 5.11s teardown test_backward_compatibility/test_select_aggregate_alias_column.py::test_select_aggregate_alias_column 5.10s teardown test_build_sets_from_multiple_threads/test.py::test_set 5.08s call test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 5.04s teardown test_allowed_client_hosts/test.py::test_denied_host 4.85s teardown test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Ordinary] 4.84s teardown test_backward_compatibility/test_cte_distributed.py::test_cte_distributed 4.64s teardown test_broken_part_during_merge/test.py::test_merge_and_part_corruption 4.64s call test_build_sets_from_multiple_threads/test.py::test_set 4.63s call test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 4.57s teardown test_buffer_profile/test.py::test_default_profile 4.51s teardown test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_soft_limit_defined_50 4.29s teardown test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 4.23s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 4.19s teardown test_backward_compatibility/test_functions.py::test_string_functions 4.16s teardown test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column 4.13s call test_compatibility_merge_tree_settings/test.py::test_check_projections_compatibility 4.13s call test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Ordinary] 4.04s call test_alter_codec/test.py::test_alter_codec_pk 3.99s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 3.98s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 3.91s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 3.86s call test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[uuid] 3.86s call test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[database] 3.80s call test_attach_table_normalizer/test.py::test_attach_substr_restart 3.75s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 3.75s call test_alternative_keeper_config/test.py::test_create_insert 3.74s call test_backup_restore_on_cluster/test.py::test_replicated_table 3.74s call test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 3.71s setup test_custom_settings/test.py::test_custom_settings 3.70s call test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 3.66s teardown test_concurrent_queries_for_all_users_restriction/test.py::test_exception_message 3.64s call test_broken_part_during_merge/test.py::test_merge_and_part_corruption 3.64s teardown test_alter_codec/test.py::test_alter_codec_pk 3.61s call test_alter_codec/test.py::test_alter_codec_index 3.48s call test_access_for_functions/test.py::test_access_rights_for_function 3.43s call test_backup_restore_on_cluster/test.py::test_system_functions 3.32s call test_backup_restore_on_cluster/test.py::test_projection 3.28s call test_backup_restore_on_cluster/test.py::test_system_users 3.25s call test_config_substitutions/test.py::test_allow_databases 3.14s call test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_50 3.14s call test_backup_restore_on_cluster/test.py::test_replicated_table_after_alters 3.10s call test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 3.04s teardown test_backward_compatibility/test_convert_ordinary.py::test_convert_ordinary_to_atomic 2.98s call test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default 2.96s call test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication 2.81s call test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 2.79s call test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_1 2.77s call test_backup_restore_on_cluster/test.py::test_file_deduplication 2.62s call test_compression_nested_columns/test.py::test_nested_compression_codec 2.56s teardown test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts 2.55s call test_aggregation_memory_efficient/test.py::test_remote 2.44s call test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 2.37s call test_backup_restore_on_cluster/test.py::test_empty_replicated_table 2.35s call test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_soft_limit_defined_50 2.34s call test_config_substitutions/test.py::test_include_config 2.34s teardown test_cluster_discovery/test_password.py::test_connect_with_password 2.28s call test_config_substitutions/test.py::test_config 2.25s call test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 2.15s call test_concurrent_queries_for_all_users_restriction/test.py::test_exception_message 2.09s teardown test_cleanup_after_start/test.py::test_old_dirs_cleanup 2.03s teardown test_backup_restore_on_cluster/test.py::test_system_users 2.01s call test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 1.96s call test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_default 1.87s call test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column 1.83s teardown test_backup_restore_on_cluster/test.py::test_required_privileges 1.79s teardown test_backup_restore_on_cluster/test.py::test_file_deduplication 1.73s teardown test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 1.68s teardown test_backup_restore_on_cluster/test.py::test_mutation 1.67s call test_backward_compatibility/test_select_aggregate_alias_column.py::test_select_aggregate_alias_column 1.63s teardown test_backup_restore_on_cluster/test.py::test_replicated_database_async 1.59s call test_cluster_discovery/test_password.py::test_connect_with_password 1.59s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 1.58s teardown test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[database] 1.58s teardown test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 1.58s teardown test_backup_restore_on_cluster/test.py::test_replicated_database 1.55s teardown test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 1.53s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 1.53s teardown test_backup_restore_on_cluster/test.py::test_projection 1.53s teardown test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[uuid] 1.49s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] 1.48s teardown test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 1.48s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 1.43s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 1.43s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 1.43s teardown test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 1.43s teardown test_backup_restore_on_cluster/test.py::test_empty_replicated_table 1.43s teardown test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 1.43s teardown test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 1.43s teardown test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup 1.40s call test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 1.38s teardown test_backup_restore_on_cluster/test.py::test_system_functions 1.38s teardown test_backup_restore_on_cluster/test.py::test_shutdown_waits_for_backup 1.38s teardown test_backup_restore_on_cluster/test.py::test_replicated_table 1.38s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_after_alters 1.38s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 1.38s teardown test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 1.34s teardown test_attach_table_normalizer/test.py::test_attach_substr_restart 1.33s teardown test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 1.31s teardown test_async_metrics_in_cgroup/test.py::test_user_cpu_accounting 1.24s call test_custom_settings/test.py::test_custom_settings 1.11s call test_allowed_client_hosts/test.py::test_allowed_host 1.09s teardown test_access_for_functions/test.py::test_ignore_obsolete_grant_on_database 0.95s call test_cluster_all_replicas/test.py::test_cluster 0.94s call test_config_substitutions/test.py::test_config_merge_from_env_overrides 0.91s call test_config_substitutions/test.py::test_config_from_env_overrides 0.90s call test_attach_table_normalizer/test.py::test_attach_substr 0.90s call test_cluster_all_replicas/test.py::test_global_in 0.86s call test_authentication/test.py::test_authentication_pass 0.75s call test_backward_compatibility/test_functions.py::test_string_functions 0.61s call test_allowed_client_hosts/test.py::test_denied_host 0.53s call test_compatibility_merge_tree_settings/test.py::test_config_overrides_compatibility 0.53s call test_backward_compatibility/test_cte_distributed.py::test_cte_distributed 0.49s call test_buffer_profile/test.py::test_buffer_profile 0.48s call test_buffer_profile/test.py::test_default_profile 0.48s call test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 0.43s call test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 0.43s call test_authentication/test.py::test_authentication_fail 0.22s call test_async_metrics_in_cgroup/test.py::test_user_cpu_accounting 0.22s call test_async_metrics_in_cgroup/test.py::test_normalized_user_cpu 0.22s call test_async_metrics_in_cgroup/test.py::test_system_wide_metrics 0.00s teardown test_alter_codec/test.py::test_alter_codec_index 0.00s setup test_backup_restore_on_cluster/test.py::test_empty_replicated_table 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge 0.00s setup test_backup_restore_on_cluster/test.py::test_tables_dependency 0.00s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[uuid] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def 0.00s setup test_check_table/test.py::test_check_replicated_table_simple[-_0] 0.00s setup test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty 0.00s setup test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def 0.00s teardown test_config_substitutions/test.py::test_config 0.00s setup test_check_table/test.py::test_check_replicated_table_corruption[-_0-.bin] 0.00s teardown test_backward_compatibility/test_functions.py::test_aggregate_states 0.00s setup test_cgroup_limit/test.py::test_cgroup_cpu_limit 0.00s setup test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] 0.00s teardown test_alter_on_mixed_type_cluster/test.py::test_alter_on_cluter_non_replicated 0.00s setup test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes 0.00s teardown test_config_substitutions/test.py::test_allow_databases 0.00s setup test_alter_codec/test.py::test_alter_codec_pk 0.00s setup test_backup_restore_on_cluster/test.py::test_system_functions 0.00s setup test_async_metrics_in_cgroup/test.py::test_system_wide_metrics 0.00s teardown test_compatibility_merge_tree_settings/test.py::test_check_projections_compatibility 0.00s setup test_backup_restore_on_cluster/test.py::test_mutation 0.00s teardown test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Atomic] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database_async 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[database] 0.00s setup test_check_table/test.py::test_check_normal_table_corruption[] 0.00s setup test_cluster_all_replicas/test.py::test_skip_unavailable_replica[two_shards_three_nodes] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert 0.00s setup test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[two_shards_three_nodes] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database 0.00s teardown test_check_table/test.py::test_check_all_tables 0.00s setup test_config_substitutions/test.py::test_config_from_env_overrides 0.00s teardown test_allowed_client_hosts/test.py::test_allowed_host 0.00s setup test_backup_restore_on_cluster/test.py::test_file_deduplication 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster 0.00s setup test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Ordinary] 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster 0.00s teardown test_cgroup_limit/test.py::test_cgroup_cpu_limit 0.00s setup test_backup_restore_on_cluster/test.py::test_required_privileges 0.00s setup test_backup_restore_on_cluster/test.py::test_projection 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table_after_alters 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts 0.00s teardown test_check_table/test.py::test_check_replicated_table_corruption[-_0-.bin] 0.00s setup test_compatibility_merge_tree_settings/test.py::test_config_overrides_compatibility 0.00s setup test_allowed_client_hosts/test.py::test_denied_host 0.00s setup test_backup_restore_on_cluster/test.py::test_keeper_value_max_size 0.00s setup test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_50 0.00s teardown test_buffer_profile/test.py::test_buffer_profile 0.00s setup test_cluster_all_replicas/test.py::test_global_in 0.00s setup test_cluster_all_replicas/test.py::test_skip_unavailable_replica[one_shard_three_nodes] 0.00s teardown test_authentication/test.py::test_authentication_fail 0.00s teardown test_check_table/test.py::test_check_normal_table_corruption[] 0.00s setup test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[one_shard_three_nodes] 0.00s setup test_backup_restore_on_cluster/test.py::test_get_error_from_other_host 0.00s setup test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica 0.00s setup test_alter_on_mixed_type_cluster/test.py::test_alter_replicated_on_cluster 0.00s setup test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_default 0.00s setup test_buffer_profile/test.py::test_default_profile 0.00s teardown test_cluster_all_replicas/test.py::test_cluster 0.00s setup test_backup_restore_on_cluster/test.py::test_system_users 0.00s teardown test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[one_shard_three_nodes] 0.00s teardown test_config_substitutions/test.py::test_config_multiple_zk_substitutions 0.00s setup test_backup_restore_on_cluster/test.py::test_shutdown_waits_for_backup 0.00s setup test_backup_restore_on_cluster/test.py::test_replicated_table 0.00s setup test_attach_table_normalizer/test.py::test_attach_substr_restart 0.00s teardown test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default 0.00s setup test_config_substitutions/test.py::test_include_config 0.00s setup test_config_substitutions/test.py::test_config 0.00s teardown test_async_metrics_in_cgroup/test.py::test_normalized_user_cpu 0.00s setup test_backup_restore/test.py::test_restore 0.00s setup test_backward_compatibility/test_functions.py::test_string_functions 0.00s setup test_backup_restore/test.py::test_replace_partition 0.00s teardown test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible 0.00s teardown test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_limit_reached 0.00s setup test_authentication/test.py::test_authentication_pass 0.00s teardown test_backup_restore/test.py::test_attach_partition 0.00s setup test_async_metrics_in_cgroup/test.py::test_user_cpu_accounting 0.00s setup test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_1 0.00s teardown test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[two_shards_three_nodes] 0.00s teardown test_cluster_all_replicas/test.py::test_skip_unavailable_replica[one_shard_three_nodes] 0.00s setup test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible 0.00s teardown test_attach_table_normalizer/test.py::test_attach_substr 0.00s setup test_config_substitutions/test.py::test_config_merge_from_env_overrides 0.00s setup test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_limit_reached 0.00s teardown test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_1 0.00s setup test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_soft_limit_defined_50 0.00s teardown test_cluster_all_replicas/test.py::test_global_in 0.00s setup test_access_for_functions/test.py::test_ignore_obsolete_grant_on_database 0.00s setup test_config_substitutions/test.py::test_config_multiple_zk_substitutions 0.00s teardown test_access_for_functions/test.py::test_access_rights_for_function 0.00s teardown test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_50 0.00s teardown test_async_metrics_in_cgroup/test.py::test_system_wide_metrics 0.00s teardown test_backup_restore/test.py::test_replace_partition 0.00s teardown test_config_substitutions/test.py::test_config_from_env_overrides 0.00s teardown test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_default 0.00s teardown test_config_substitutions/test.py::test_config_merge_from_env_overrides =========================== short test summary info ============================ PASSED test_alter_codec/test.py::test_alter_codec_index PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_default PASSED test_access_for_functions/test.py::test_access_rights_for_function PASSED test_allowed_client_hosts/test.py::test_allowed_host PASSED test_allowed_client_hosts/test.py::test_denied_host PASSED test_alter_codec/test.py::test_alter_codec_pk PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_1 PASSED test_config_substitutions/test.py::test_allow_databases PASSED test_backup_restore/test.py::test_attach_partition PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_defined_50 PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-False] PASSED test_config_substitutions/test.py::test_config PASSED test_access_for_functions/test.py::test_ignore_obsolete_grant_on_database PASSED test_config_substitutions/test.py::test_config_from_env_overrides PASSED test_cluster_all_replicas/test.py::test_cluster PASSED test_config_substitutions/test.py::test_config_merge_from_env_overrides PASSED test_check_table/test.py::test_check_all_tables PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[http-True] PASSED test_concurrent_threads_soft_limit/test.py::test_concurrent_threads_soft_limit_limit_reached PASSED test_backup_restore/test.py::test_replace_partition PASSED test_authentication/test.py::test_authentication_fail PASSED test_attach_table_normalizer/test.py::test_attach_substr PASSED test_authentication/test.py::test_authentication_pass PASSED test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_default PASSED test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[one_shard_three_nodes] PASSED test_config_substitutions/test.py::test_config_multiple_zk_substitutions PASSED test_concurrent_threads_soft_limit/test.py::test_use_concurrency_control_soft_limit_defined_50 PASSED test_backup_restore_on_cluster/test.py::test_async_backups_to_same_destination[native-True] PASSED test_attach_table_normalizer/test.py::test_attach_substr_restart PASSED test_config_substitutions/test.py::test_include_config PASSED test_backup_restore/test.py::test_restore PASSED test_cluster_all_replicas/test.py::test_error_on_unavailable_replica[two_shards_three_nodes] PASSED test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Atomic] PASSED test_cluster_all_replicas/test.py::test_global_in PASSED test_check_table/test.py::test_check_normal_table_corruption[] PASSED test_backup_restore_on_cluster/test.py::test_backup_restore_on_single_replica PASSED test_attach_partition_with_large_destination/test.py::test_attach_partition_with_large_destination[Ordinary] PASSED test_alter_on_mixed_type_cluster/test.py::test_alter_on_cluter_non_replicated PASSED test_backup_restore_on_cluster/test.py::test_different_tables_on_nodes PASSED test_cluster_all_replicas/test.py::test_skip_unavailable_replica[one_shard_three_nodes] PASSED test_backup_restore_on_cluster/test.py::test_empty_replicated_table PASSED test_alter_on_mixed_type_cluster/test.py::test_alter_replicated_on_cluster PASSED test_buffer_profile/test.py::test_buffer_profile PASSED test_buffer_profile/test.py::test_default_profile PASSED test_backup_restore_on_cluster/test.py::test_file_deduplication PASSED test_backward_compatibility/test_insert_profile_events.py::test_new_client_compatible PASSED test_backup_restore_on_cluster/test.py::test_get_error_from_other_host PASSED test_backward_compatibility/test_insert_profile_events.py::test_old_client_compatible PASSED test_async_connect_to_multiple_ips/test.py::test PASSED test_backup_restore_on_cluster/test.py::test_keeper_value_max_size PASSED test_aggregation_memory_efficient/test.py::test_remote PASSED test_alternative_keeper_config/test.py::test_create_insert PASSED test_backup_restore_on_cluster/test.py::test_mutation PASSED test_compatibility_merge_tree_settings/test.py::test_check_projections_compatibility PASSED test_compatibility_merge_tree_settings/test.py::test_config_overrides_compatibility PASSED test_backup_restore_on_cluster/test.py::test_projection PASSED test_cluster_all_replicas/test.py::test_skip_unavailable_replica[two_shards_three_nodes] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database PASSED test_broken_part_during_merge/test.py::test_merge_and_part_corruption PASSED test_cgroup_limit/test.py::test_cgroup_cpu_limit PASSED test_build_sets_from_multiple_threads/test.py::test_set PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_async PASSED test_backward_compatibility/test_functions.py::test_aggregate_states PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_compare_parts PASSED test_custom_settings/test.py::test_custom_settings PASSED test_backward_compatibility/test_select_aggregate_alias_column.py::test_select_aggregate_alias_column PASSED test_concurrent_queries_for_all_users_restriction/test.py::test_exception_message PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[database] PASSED test_backup_restore_on_cluster/test.py::test_replicated_database_with_special_macro_in_zk_path[uuid] PASSED test_backup_restore_on_cluster/test.py::test_replicated_table PASSED test_backward_compatibility/test_cte_distributed.py::test_cte_distributed PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_after_alters PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_bigger_cluster PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_restored_into_smaller_cluster PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_def PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_insert PASSED test_backup_restore_new/test_shutdown_wait_backup.py::test_shutdown_wait_backup PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_not_synced_merge PASSED test_backup_restore_on_cluster/test.py::test_replicated_table_with_uuid_in_zkpath PASSED test_backup_restore_on_cluster/test.py::test_required_privileges PASSED test_cluster_discovery/test_password.py::test_connect_with_password PASSED test_backup_restore_on_cluster/test.py::test_shutdown_waits_for_backup PASSED test_cancel_freeze/test.py::test_cancel_backup PASSED test_backup_restore_on_cluster/test.py::test_system_functions PASSED test_backward_compatibility/test_convert_ordinary.py::test_convert_ordinary_to_atomic PASSED test_backup_restore_on_cluster/test.py::test_system_users PASSED test_backup_restore_on_cluster/test.py::test_table_in_replicated_database_with_not_synced_def PASSED test_backup_restore_on_cluster/test.py::test_table_with_parts_in_queue_considered_non_empty PASSED test_backup_restore_on_cluster/test.py::test_tables_dependency PASSED test_cleanup_after_start/test.py::test_old_dirs_cleanup PASSED test_compression_nested_columns/test.py::test_nested_compression_codec PASSED test_azure_blob_storage_zero_copy_replication/test.py::test_zero_copy_replication PASSED test_backward_compatibility/test_normalized_count_comparison.py::test_select_aggregate_alias_column PASSED test_check_table/test.py::test_check_replicated_table_corruption[-_0-.bin] PASSED test_check_table/test.py::test_check_replicated_table_simple[-_0] PASSED test_backward_compatibility/test_vertical_merges_from_compact_parts.py::test_vertical_merges_from_compact_parts SKIPPED [1] test_async_metrics_in_cgroup/test.py:64: Disabled for sanitizers SKIPPED [1] test_async_metrics_in_cgroup/test.py:78: Disabled for sanitizers SKIPPED [1] test_async_metrics_in_cgroup/test.py:44: Disabled for sanitizers SKIPPED [1] test_backward_compatibility/test_functions.py:164: The test is slow in builds with sanitizer ================== 96 passed, 4 skipped in 427.10s (0:07:07) ===================